21st Feb 2022

We recently spoke about WASM packages. We have fpm packages that will be used to distribute static assets. We want FPM packages for font support. We also need FPM packages to distribute syntax definitions and syntax themes. We can maybe create FPM package to distribute SQLite or CSV data files.

In many cases the advantage of creating FPM package exists, even when the end users of the thing being distribuetd, font, sqlite, syntax definition, is not going to use FPM itself.

Say you have created a font, you would want to distribute the font to both general audiance, and to people who want to use your font in a FPM package. In such a case it would make sense for you to create FPM package as FPM package can become the website for you, tomorrow when we have FPM repo in place you can easily monetise and otherwise control the distribution of your font.

20th Feb 2022

Once translation feature is polished, and special packages are imeplemented, processors are done, we have to start thinking about FPM Repo. It is planned but no work has started on it. FPM Repo is quite ambitious, we want to do a lot but how do we begin?

One option is we become target for static hosting. FPM is going to have dynamic features, but we have not yet built them, so we can at least do static hosting. Once static fpm repo is done we can look at dynamic stuff (fpm processors running on page request time instead of publish time) and CR/online editing stuff (not sure in which order, static -> dynamic -> CR seems logical because dynamic does not have frontend UI, is purely backend work, and also maybe we will build CR/editor etc using dynamic FPM).

19th Feb 2022

We currently ship 3-4 processors as part of FPM. These are part of FPM source code, and are implemented in Rust. We also ship a few “host methods”, which FTD can call using message-host event handlers. These are implemented in JavaScript.

Having only a few such processors and methods is obviously not great because people are going to have arbitrary needs, and we can not hope to anticipate all of them and ship all such methods. We have to make it such that people can write their own.

9th Feb 20222

For static assets we have to solve three problems.

4th Feb 2022

So Shobhit has implemented the “main” proposal. What we are trying to avoid is imports at top of each file. With main we manage that. But now we lose discoverablity, where does a symbol come from. In main proposal the fact author is coming from foo.com/book-theme is not obvious as you are writing:

include: * would be default. We have can have include: defaults to only import default ones. We can also include individual patterns explicitly.

Note: exports are not considered part of package interface.

3rd Feb 2022

I was going through MagicBell and I quite love them. It gave me the idea that people today almost universally hate the email subscription dialog, and yet universally are driven to clicking that red icon.

We will ask people to keep only some meaningful number of items in news, say 10, and keep deleting the old ones. Or we can keep the entire archive and only consider the first 10 when showing the notifications. We can also show all the news items in /FPM/news/ page.

We will create a package: fifthtry.github.io/notifications which will act as the widget, read the fpm.news and show. We will have a message-host function to store the latest notification created-on in a cookie, and on page load the value of the cookie would be made available on fpm.latest-news-read variable, so any news items later than fpm.latest-news-read will be shown as unread.

We will mark the news as read when the notification icon is clicked.

To impelment this we need date-time type ideally. Else if we store created-on as string, we still need > check. We also need a way to get first item from a list. To show the dot on the closed icon state, we need check of cookie value vs created-on of latest news. created-on of latest news can be stored as a another fpm variable for convenience till we get first item of a list support. Once the UI is open, we will want to show what all is new, so we will def need string comparison operator, >.

Currently FPM is fully distributed, each FPM package is published on a URL controlled by the owner of the package. To aid discoverablity tt would be cool if there was a package registry though, a central place where you can quickly discover packages.

Is this going against our decentral design? Not really. FPM does not really depend on the registry unlike other package managers. You can depend on any package and if some package is removed from package registry, all it will lose is a listing one some site. The package itself would be still available and anyone who want to use the package can still use it.

Registry is just a handy place to quickly find themes, components etc, we can easy show all (most) packages that implement a given package interface. We can even show some stats like how many packages depend on any given package, which can be a good proxy for people who want to create a dependency on that package, a package that is quite popular for example maybe safer to use than a package that just came up yesterday and no one is using it.

We will make fpm.dev the “fpm central” or “fpm package repository”, which means eventually fpm.dev will have dynamic stuff. Can we build the dynamic stuff using our proposed FPM Web Framework? Time will tell. We will not block on it. FPM package repository can tomorrow become an open source real world showcase of fpm web framework when it is in place tho.

2nd Feb 2022

THe idea is we will create package.zip in .build folder whenever we do fpm build. Currently the zip file we download is generated by Github and contains the source of entire repo.

In the package.zip we will include the entire repo.

In this case the repo has a single package, and the build will simply include all the files of the package minus the files ignored by fpm build (we auto ignore some files like .git etc, and we allow FPM.ftd to specify other files and patters to ignore).

Consider a repo with multiple FPM packages:

In this case when we are are calling fpm build we have to cd into one of the two folders, and each en and hi will get their own .build folders containging their own package.zip files.

The zip will still include all the files in the repo based on previous ignore logic. We will introduce a new ignore heuristic, if you come across a file that belongs to another FPM package, auto ignore it as well. So when building package.zip for en folder, code.rs will be included but not hi folder.

A package can opt out of auto-ignore-other-packages-in-zip heuristic by overwriting this key to false in FPM.ftd. This heuristic is only to reduce the package.zip size.

The package.zip will also contain FPM.manifest.ftd file which will be currently contain only one setting: the location of FPM.ftd file that should be the starting point. Since if we have disabled auto-ignore-other-packages-in-zip heuristic, en/.build/package.zip will contain both en and hi folders, and without the manifest file we do not know where the FPM.ftd file is so we need the manifest file.

29th Jan 2022

When creating a new package we have to write, e.g.:

Which is annoying. We need the publish URL, and that is good, we are going to share it everywhere. But we need zip also, and its only used by fpm for internal purpose (downloading the package). But writing the zip key and reading it is both noise for humans.

My first thought was we auto generate the zip URL by parsing the package name, which would be fragile and won’t work if someone is using custom domain etc. Then it occured to me that we can create the zip file and upload it as part of the build itself, so the zip file url will always be <package-name>/package.zip.

It has one positive aspect: zip url as defined currently can diverge from the published package. People may see one thing on package url but may download something else, the latest code on main. The build may not have happened on subsequent commits, so things can diverge. The zip as part of package ensures we always get the source of what we see.

This can arguably be the downside as well, if there is something wrong with the package, one can not just fix it by committing on main, one now have to figure out how to update the zip. I am not sure what will be the cause of this tho.

Overall I think we should go ahead with this proposal.

Was wondering how would we implement this site. They have a single page, and all header links simply scroll to different portions of the same page. The scroll behaviour is also smooth, not abrupt.

We can implement the abrupt scroll behaviour currently, we give each heading a unique id, or authors can specify their own id. So link targets can be #foo. But this will jump abruptly.

So we will give the following fpm host function:

This will use the “smooth” behaviour.

We should eventually be able to do:

Me and Shobhit were wondering how would we fix markdown styling. Currently we have a css file that contains styling that gets applied to markdown generated html and currently there is no way to update that CSS.

We want any theme to be able to update the markdown styling. And yet we do not want to let anyone modify CSS, we consider CSS a liability, an escape hatch, which will hamper our plans to write non HTML based rendering for FTD.

When we parse markdown we get elements like heading, paragraph, link, image, tables, and lists. We eventually realised if we had a syntax like this:

We can solve our problems. ftd.markdown would be a new ftd kernel element which will take a bunch of ftd.ui constructors, which will then be invoked for every heading, paragraph etc that we encounter in the markdown.

This way any theme can create their own markdown wrapper component and pass it all the component constructors, and completely control the rendering.

26th Jan 2022

Recently discussed “Thought: Font/Asset Packages”, and we had a need for packages that can be used to distribute fonts, and other assets like images. When designing some theme we realize we have to give users ability to modify the syntax definitions, we currently ship with whatever is shipped with syntect and a few extra syntax definitions in ftd repo.

We have to allow people to define both extra language support and theme support. Also we have to ship language and themes as FPM Packages, so they can added as dependencies. We would need some meta data in FPM Package to indicate they are special theme or syntax definition package. We can do it by package interface, if a package is defined to implement a package interface, e.g. fpm/syntax of fpm/syntax-theme, we would look for special files in such packages that are dependency of current package and make them available to ftd.

Currently when you see -- import: foo, foo can either be package name or alias, or foo can be a module (ftd file) in current package. What if we stop doing the later, and the only way to import foo.ftd in current package was: -- import: package:foo?

We can go even one step further, and allow package name itself to be aliased (if needed), and always use <current-package–name:module-name> when importing.

So both? I tend to prefer one. Rust does crate. But they also have to rename crate to any name, and I tend to use that in all my Rust code instead of crate.

21st Jan 2022

Shobhit has been implementing some release automatons off late. Now our Github Action creates binaries for the three supported platforms (we only run our unit tests on Linux, so technically Windows is not completely supported right now, but we still want to, if you come across any issue please let us know and we will resolve asap). The workflow also creates a release.

We have a script, install-fpm.sh, which we recommend you use for you CI systems for installing latest release of fpm. It supports both Linux and Mac. Also it lets you chose if you want to use the latest pre-release or the latest release of fpm.

You can use install-fpm.sh --pre-release to get the pre-release. Without the argument it installs the latest release.

FPM.dev, ftd.dev and a bunch of FifthTry sites use fpm in --pre-release mode so as to catch bugs and regressions early.

It has a few issues:

1. We have both title and URL as compulsory parameters, we want to allow people to create TOC entries that are not yet linked with any URL, and show them in TOC without a link, for creating entries for chapters or topics that are coming soon.
2. We want to be able to specify src: to be used as icon for some entry.
3. We want to add “separators” in TOC
4. We want to be able to use “part headings”, these are not links, but are used to group a bunch of chapters together.
5. We want people use font-icon like fonts.
6. More compact link specifications when it makes sense.

Title now comes in first line. If there is space you can put title and URL in same line. url, src, font-icon, disabled etc other meta data you can pass to an entry in toc.

Nesting is still there, by two spaces. You can use free spaces before and after links, but not between the properties.

14th Jan 2022

So yesterday we settled with this as solution for our prelude problem:

After a good night sleep I feel we can evolve this further. The thing is we did not discuss a “default” exports for every “package interface”, nor are we using the planned implements syntax yet. The idea is when you import a dependency, you may want to use it only because it implements the package interface of a package you are interested in. Say you want something that implements fpm.dev/blog, and you found Jay/Fancy-Blog-Theme. Now Jay/Fancy-Blog-Theme may implement more than just fpm.dev/blog. It may implement other packages. Also Jay/Fancy-Blog-Theme may define some components that are not part of fpm.dev/blog, which means if you used them, switching to another theme would become hard as that other theme may not implement Jay/Fancy-Blog-Theme.

So we do not want to access all modules and symbols of a package, we want to constraint what modules and symbols are “visible” to us. We can do that using proposed implements syntax:

Using implements does three things: 1. ensures fifthtry.github.io/Jupiter-theme has modules and compatible symbols as defined in fpm.dev/blog, 2. any attempt to access any extra modules are symbols, that are not part of fpm.dev/blog lead to error, and 3. makes fpm.dev/blog an alias for fifthtry.github.io/Jupiter-theme, so if any document has imported fpm.dev/blog, they will be getting fifthtry.github.io/Jupiter-theme instead.

We can use implements more than once on a dependency.

Now if you look at the two code snippets, its obvious that book: j.book is conflicting with implements. We are defining the same thing twice. We are going to have to definitely use the implements clause. But also the book: j.book.

Can we somehow do it in one go? This is not just a question of reducing the amount of things you have to write, its also about creating a consistency. Consider if a package can define default imports / “main” modules:

Meaning if any ftd file now on-wards refers to blog, they will be getting j.blog, i.e. j.blog is auto imported in every ftd file. And we do not even have to write the explicit lines, we follow implements, and look at main for the package being implemented, and auto import the main modules.

So while it obviously is a lot of “magic”, we are getting a consistency, everyone refers to fpm.dev/blog.author with the alias author, even when we are using different theme. So modifying the theme becomes easy. All the main imports would get well known soon etc.

What will happen when say both fpm.dev/blog and fpm.dev/book have defined say a module named author?

In this case we can go on and define things explicitly:

Here we are saying author will refer to j.author, the default, and k has given up author and used book-author name.

We can even support this syntax:

Here we are explicitly renaming author to book-author, and we do not have to care about the alias.

Closing note: this is a lot of “magic” indeed, but consider that in scientific computing world, machine learning world etc, using well defined aliases for common Python modules, e.g. np for numpy, pd for panda, plt for matplotlib are almost universally recognizable. We are creating few such aliases, book, post, etc, which we hope to similarly become universally recognizable, and so this magic is acceptable. Creating a consistent naming, so any ftd document you come across in wild can be easily copy-paste-able in your project is easily achieved by this. You will always know what book is if you come across it anywhere, and can quickly pick from a huge number of themes that are implementing book.

We have to first implement simple things like “object constructor” in FTD. This is the core to how we pass data from FTD to “host”. We similarly need a “special variable” $VALUE, which will be available only in $on-change$ handlers, so we can do $on-change$: $query = $VALUE, assuming we already have a variable defined: -- optional string query:. These are easy bits.

Then we have a harder bit. In FTD the DOM structure is rendered by Rust code, and from front-end we can only modify some variables, variables that show-hide an existing DOM node, or change some attribute of existing DOM node. This is why we have to make every field conditional individually, we have to update our JS to change the DOM node.

What we have to do now is an ability to generate DOM node in front-end now. Search is going to fetch search result from some service, and we have to create a child corresponding to each search result. We can do a hacky thing, pre-create say 10 DOM nodes, because we know the page is going to have 10 search results only, but this is not a great answer, say consider if we had tree, with up-to 4 level, we will have to pre-create 10 * 10 * 10 * 10 DOM nodes.

So me and Arpita decided we create a proper support for creating DOM based on component definitions from front-end. We will pass a specification of components that are needed dynamically (we do a dynamic variable dependency analysis already). And then we will write some JavaScript code in ftd.js to create DOM based on the component spec, and the data (it will happen every time the data changes).

We already have a bunch of JS logic, this change will introduce quite a bit more, still quite low compared to other JS library standards, our current ftd.js is only 3.3KB gzipped, it will barely increase it to 10KB at worst. This will increase the amount of JSON we include in page, as now the specification of all the components used dynamically will also be included. Our compresses index.html for example is only 12KB (which includes JS/CSS etc everything), so we still have quite a small built size.

We have done some interesting parsing for what we call “ftd markup”, we will have to port that code over as well. Which means we have to have a full markdown parser in JavaScript, which will further increase the size.

We can do this in a conditional way, only include markup/markdown parser if one of the dynamically called components need it. Or only include dynamic component rendering part of JS if there are at least one dynamic component. But even in worst case I do not see our compressed, everything included HTML going above 20-30KB, which is quite acceptable.

One very interesting consequence of this feature would be that FTD would become a general purpose UI library that can be used with any JS project. We would be almost like Elm, like Elm has “ports”, we will have message-host to send data from FTD to JS, and ftd.set_var family of functions to set FTD data from JS.

This means you can tomorrow write full web applications like Twitter and GMail with FTD. Your JS will have no idea about DOM, your JS will simply set FTD variables, and FTD will take care of rendering them, and will call your JS from time to time, based on user events etc via message-host.

Was reading Rethinking errors, warnings, and lints, and I quite liked it. We should have a strict mode in ftd and a regular mode. In strict mode we fail on any thing that maybe wrong, eg un-used import, variables or components with improper case, un-used component arguments and so on. We will try to be as strict as possible. Maybe even bad formatting should be considered an error in strict mode. All arguments in component definition must come before the properties for example.

We will then have fpm build not use the strict mode, so authors can quickly try things out. And finally we should have fpm build --release, which will use ftd in strict mode, forcing people to ensure all sloppy-ness is fixed before something is published.

We can also have fpm check which will test for all warnings, and can be used as part of pre-commit hooks.

We used to depend on git when doing validation of fpm.package.zip URL. This failed on Ganesh’s Windows machine as he does not have git cli installed. We learned that Github Desktop does not install git cli on Windows.

So Shobhit converted our code to use git2 (libgit) as a Rust crate dependency, so we do not depend on external binary. I further ensured that git2 is compiled with bundled SSL and LibGit libraries so our CLI becomes more self contained.

While I was doing that I realised the reason we use git is currently not strong enough, so Arpita removed git2.

In future we may be bringing git2 dependency back though, when fpm becomes a full blown “editor” as well. We want fpm to one day be only binary authors need, it will be the editor, so you do not say SublimeText etc, it will have git internally so you can clone and commit from fpm editor itself, we will make fpm run as a web server, and maybe a side by side preview with auto reload.

Integrated Authoring Environment it would be, IAE.

We used to have a special module named fpm-ui with a bunch of variables for translation etc. It used to also contain ftd-ui.mobile and fpm-ui.dark-mode flags as well. We also had another special module fpm, with other set of similar variables.

It was an implementation detail, there was some feature missing in FTD (ability to refer to record fields defined in another module), and Arpita did not like “polluting” fpm with such variables which would tomorrow become record fields, so she created fpm-ui.

Since then Arpita has implemented the fix, you can now do foo.bar.baz and foo maybe another module, foo.bar an instance of some record, and .baz a field on that record. So she cleaned things up by removing fpm-ui and moving all the variables to fpm.

We have not yet properly documented those variables which we will be doing soon.

13th Jan 2022

We had a feature called prelude. You can define a file FPM/prelude.ftd, and this file is prepended to the top of every .ftd file in the package when its being evaluated by FTD.

I was showing a demo of FPM against Docusaurus to a friend today, and I feel we have done quite a decent job. What they have over us right now is Versioning and Search.

On versioning we have already detailed the plans on how we will build a much superior version than what is out there using our tracking feature. But we have never discussed how can we implement search.

How Docusaurus implements search is by embedding Algolia’s DocSearch widget. DocSearch provides an integrated search experience, a library to fetch data from backend, in this case Algolia, and a UI with autocomplete, Command Palette interface and so on. It’s great as as drop in implementation.

We can not really use DocSearch though. We can not use any JavaScript library for UI because our UI is in FTD. So how would we implement search?

Let’s assume for a moment that we are still going to use Algolia for indexing, core search feature. There are many alternatives, MeiliSearch’s, typesence, even ElasticSearch, PostgreSQL Full Text Search and so on. It’s not interesting what we are using.

So lets say we are using one of these, and for purpose of this write-up lets say there is a function search(query) available in JavaScript that can initiate the search, and lets say it calls found_results(results) every time the result is ready. Let’s assume these two functions do absolutely nothing with DOM.

2nd Jan 2022

Still can’t believe 22 is here! So I have been thinking about fpm update recently and came across nostr that implements something very close to what I had in mind.

I kind of consider nostr design the right design, good balance between centralised (prone to attacks, censorship etc) and “pure” peer to peer (complex, less reliable etc). Maybe there are issues, but to me it sounds good.

So we know how to distribute packages. What is missing is package identity. Package names depend on DNS if you buy your own domain, and with others, eg github if you host on <username>.github.io etc. So how do we ensure you really own the package identity? That you can move from .github.io to something else without losing people who may be interested in your content.

Traditional way requires you to do a redirect from old to new. But this relies on old service still running, often companies come and go, domain can expire or sold to someone with no interest in honouring the old contracts.

If not domain name then what? One candidate is cryptographic keys. But they are too hard, at least when you are targeting billions of publishers, like Twitter and Facebook does. Eventually your FPM packages should be your twitter/Facebook profile, but how do you manage identity?

WhatsApp etc, E2E, End To End-Encryption systems rely on some shared identity, eg a phone number of email address as a convenience, and then they generate some keys behind the scenes, you can see the fingerprint of your keys in whatsapp for example, and verify each others identity (phone number appears to be the identity from the layman’s perspective, but its backed by strong cryptographic foundations).

So let me repeat, when you are sending a message on WhatsApp, you think you are sending it to a phone number, but WhatsApp has internally exchanged some encryption keys with that contact, and its going to that key.

Let me say it in a more formal way, in PKI, whatever that might be, anyone can generate a key pair, a private key and a public key. This key pair can be considered an identity. When you send a message it goes to someone, that someone needs to be identified, and how we identify them can be considered their identity. What you can do is encrypt the message with the public key of the recipient, and only the corresponding private key can de-crypt the message.

So in this sense, a key pair can be your identity. But key pairs are just numbers, and unlike phone numbers, they are much bigger. So exchanging these numbers is not fun, it’s opaque, no one can remember them, nor can see what they are referring to if we see a huge number representing a public key in someone’s FPM.ftd file as a dependency.

So we need a way for the key pair to get an alias. The domain name is a good alias. If we tread DNS as alias, not the source of truth, things are much better. Identity moves are rare, but they do happen. People switch providers, people may want to move from Github Pages to another service, and back. Once someone starts following someone, such moves should not affect them, the followers should continue to get updates despite name changes. So how do we achieve this with PKI and aliases?

Imagine Github Pages implemented our protocol, this is how it could work. They generate a key pair for you when you sign up. This is important if we want billions of people to use services. Most people are not going to want to bother with all this. This is what WhatsApp/Telegram do to deliver strong cryptographic guarantees (at least by design/promise, implementation is another matter) to masses.

So when you create an account on any of these services, they create a new unique identity (key pair) for you. You start using that service, you gain followers, and say you get serious. This is the point you will bother with all this identity business details. Now you have say 100s of followers and you want to move to a different provider, without losing the followers (without breaking 100s/thousands of packages that depend on breaking).

At this point you will now claim your identity. You install another app, offline only for strictest guarantees, and generate your real identity, real key pair.

Now if Github Pages etc are acting in good faith, they will let everyone know that they are temporary guardians of your identity, that you have not yet claimed your identity, and the moment you do so they will cooperate. If they chose to act in bad faith, charge you money for you to claim your identity or not let you do it while initially promising that they will, then you could lose follower, but hopefully word will spread and people will shun such services.

So the temporary guardian of your identity can accept your real identity keys, and everyone who is following your via <username>.github.io now learns of your true identity as well (and that Github is an intermediary working for you for now, via you signing the Github generated keypair with your real identity).

So real identity has always been the keypair, but we would be using the domain name. We need a distributed store of keypair to DNS mapping, so you can switch the domain for your keypair. Some sort of “key server” is needed. Your followers where initially thinking that the Github generated keypair was the true identity, but since you update your identity, github will notify them about the true identity and your FPM will now store the true identity of what you are following.

We also will have to allow for “key revocation”, say one of the guardians went rogue, got bought out, hacked etc, and starts sending updates on your behalf to your followers, you had signed that key-pair with your key, so now you have to revoke that key-pair. The key-server can do that as well.

So the protocol is: in FPM.ftd you see domains. You keep a cache of domain to keypair id on your file system. Periodically you check with key-server if the domain for any keypair has changed, if so fpm commands start reporting warnings that domain entries are out of date, please update ASAP. fpm can now chose to, if you so want, start using the latest entries from key-server for the identities, so say if dependencies say you are publishing on <username>.github.io, but since key-server says domain moved to <username>.com, fpm will start fetching from <username>.com.

1st Jan 2022

Arpita has been working relentlessly on transition UI and I am quite proud of what she has done so far.

So now we are building a lot of UI that is going to be part of a website you create. The website you create using FPM/FTD can be exactly how you want it to be, you have absolute control over every pixel of the page. But then we have our UI elements, things that are needed for FPM to work correctly, FPM/info.ftd file, FPM/translation/*.ftd files and so on.

And you can totally edit those files, copy them from our source, tomorrow we may give you some cli helper for that, and edit them to your hearts desire.

But we do not believe that is ideal. We are going to create a theme, a “document interface” in our terminology, which will contain all the components needed for FPM related UI that gets included in your site.

Once that document interface is finalised, we will create a few themes implementing fpm interface, and hopefully end users can as well. License permitting you can use any of them interchangeably.

Ideally all the main themes should also implement FPM Theme document interface as well, so most theme you find for your blog, resume, book, open source project page, startup landing page etc, will come with compatible elements, but if the theme you picked for rest of your site does not come with the FPM Theme elements, then you can use another theme visually compatible with your theme, or create your own.

You have full power over entire UI, the interaction, style, and you only need to learn FTD to design those pages, FTD something that is lot easier to learn than HTML/CSS/JS for doing this kind of job. FTD that is designed for non programmers even.

So we recently introduced this command, and currently it simply re-downloads all the dependencies, which is fine for our demos and initial usability testing etc, but clearly not something we want to eventually.

We are designing FPM infra to be as “distributed” as possible, this means we can not have a central entity knowing the status of all the packages as is done by many other package managers.

We can poll, re-fetch each <dependency>/FPM.ftd, with etag handling etc, but this is still not great. How often to poll? On every fpm command invocation? Every minute/hr/day? Every time user says fpm update?

Sure. We can come up with some such scheme. But can we have some sort of middle ground? Not central, yet we do not have to poll everyone? Poll is not great, we want push, as soon as the package we are interested in is updated, say I opened my laptop for a few minutes before catching a long flight, I would want my system to automatically fetch all the latest updates. If in the last minute before I close the lid there was an update to a package, I would want that to be available to me.

Posing problem like this, no central, and yet want push, means we create a few, a few hundred, thousand maybe “central” agents. Say there is a new component fpm bastion, probably using that word incorrectly, some sort of fpm aggregator, and say each package can pick say 10 of such aggregators. People can maintain fpm aggregator lists, some will be available via fpm.dev, but say its down, under attack, sold out, there are others lists, and each package may include a list of aggregators they know of so one can look at all aggregators from all the package they currently have.

So there are so many aggregators. And they themselves are chatter boxes, gossiping with each other. For each package they keep track of the FPM.ftd file. We can always use public key infrastructure to ensure the FPM.ftd file I receive from an aggregator is trustworthy (they can lie about not giving me the latest update, but they can not modify FPM.ftd without getting trivially detected).

Each FPM package should pick a dozen or so aggregators from the list. And every time there is an update, every time fpm sync runs, the content is uploaded to fpm-repo but also sent to aggregators.

Aggregators are chattering and they will communicate to others if any change has happened. Aggregators may even be paid to give guarantees that they talk to everyone or that they have strong latency guarantees (the gap between an update happening on any aggregator in world to this aggregator telling you it has happened).

We will have to employ many techniques from distributed hash table literature and prior work. Maybe even use IPFS etc and not re-invent things.

Or we can go old fashioned, sort of reverse mail relay, DNS for any domain (or current version of FPM.ftd if its not too old) can tell you the preferred aggregator for this package. We can also do the aggregator push, like my preferred aggregators are say this list, then when I am fetching a package from your fpm-repo, I can tell the repo about that list, and tell fpm-repo to notify to one of my preferred aggregators.

Anyways, so we have a bunch of aggregators, or we have full blown IPFS infra, whatever. We will now need our fpm to have a demon that will run in background and keep fetching stuff. All of this can be done by your fpm-reader.

fpm-reader would be a service for consuming FPM packages. You can install other packages as dependency of a package you are building, but can also “subscribe” to packages via fpm reader, which can be running on your machine (as part of fpm binary maybe), or on your Heroku/DigitalOcean instance if you have devops bandwidth, or you can buy it from a SAAS provider.

If you are using a cloud fpm-reader (meaning not running on your machine), your fpm daemon will connect with fpm-reader and get the updates via a web-socket/ push mechanism.

Update: Turns out nostr describes the protocol very close to what I had in my mind.

31st Dec 2021

We have a feature that allows you to query SQLite databases. In general it is good feature, if you have SQLite database, sure, go ahead and query it. But we call the query processor package-query, because the original intent of that processor was to be able to query a special database that shows FPM package data. Our first design was to do a build-package-database that generates a sqlite file, and then run queries on that sqlite file.

The main idea was we do not want to create our own query language or API, that’s a lot of work, and new thing for people to learn, but we do want to expose rich data so people can make their documents smarter.

Turns out there is a project, GlueSQL:

GlueSQL is a SQL database library written in Rust.

It provides a parser (sqlparser-rs), execution layer, and optional storages (sled or memory) packaged into a single library.

Developers can choose to use GlueSQL to build their own SQL database, or as an embedded SQL database using the default storage engine.

This sounds like pretty much what we want, a SQL interface, but powered by our custom logic.

The main reason to use it would not be just because we don’t want sqlite dependency, or anything is wrong in sqlite, but because build is circular, you can not create package data sqlite database from data in documents because some of those data may be generated from a package query itself.

Not sure what it entails when it comes to actually implementing it, but knowing there is good starting point, we will not have to start from scratch means this solution is now at least conceptually feasible.

30th Dec 2021

Arpita has been relentless about translation support. After finishing the basic features she is now focussing on data presentation. Some of the UIs that are ready.

We have two large level approach for UI message internationalisation. First is the tried and tested, i18n approach, say we use Mozilla’s Fluent. We can define special strings we have in our source code, and we invest in comprehensive translation so no one has need to improve it (meaning it can not be done by writing FTD files, you must recompile fpm).

I am not fond of that approach, primarily because while we can do that for strings we know about, but what about themes and other kind of content people create? Say a theme for book (document interface) will have terms like “table of content”, how to centrally translate such information so that each user of that document interface in that language doesn’t have to keep deciding how to translate things.

We need to think more about this.

2 letter codes are okay, but in FTD/FPM we tend to stick with full names instead of short versions, like integer instead of int, or language instead of lang, since we believe things are written once and read many many times, and we have immense capability in getting confused. How many people can guess gu stands for Gujarati and not some African language? What have we really saved by writing gu, instead of writing “gujarati” which has to be written precisely once in the package which is probably going to have tens or thousands of words?

We have a few problems here. One we must ensure the language names are exactly the same, we all call it either “Gujarati” or “gujarati”. This is because we can then easily re-use translations etc, standardisation is good in general.

We can use the “ISO Language Name” column as shown in the Wikipedia (or find out where did they get the actual data from). They have mapped more than one names to same code in a few cases, which I feel is actually a good thing, we can be more precise in language name, honour the preference of the author, we are interested in the code, which we will get.

So this is our first decision. We will proper case also, so language: Gujarati as it is proper noun, and not language: gujarati.

Then the other problem is how to show the name of language in language detail page in a form native speakers of that language can understand, so “ગુજરાતી” we should show. But the translation status page we should show probably both ગુજરાતી and Gujarati.

So I was thinking of the two names for each language, and how both are relevant information, should we allow both somehow? Say language: Gujarati | ગુજરાતી, we will ignore the part after |, so that can be used as some sort of description about the translation.

This got me into thinking if we must have a requirement that there can be only one translation in one language, since if not this allows us to also explain what is happening, so we have syntax, but do we have a need?

This got me into thinking what if there was a book “Rust Programming Language” and someone translated it into “Rust Programming Language for Advance Haskell Programmer”, it will remain in English, but the audience has changed, advance Haskell programmers probably do not need such a laborious, long explanation, maybe their book would be just a 10 page book instead of the 100 (random guess) page current book?

After all the purpose of translation is to make some work accessible to a population who wanted to read it but could not because of language barrier. But isn’t a book written in too much (or too little) technical detail a language barrier itself? Can you read a book written for absolute beginner of a topic if you are in advanced stage, and vice versa? Can these two people really talk to each other meaningfully? Isn’t there is a need for a translation there?

And consider this also: say I am describing the strict specification of what FPM does, shouldn’t there be a document like this? Which assumes you know everything, and you are only interested in exact description of precise behaviour of what FPM does? Of course that document must exist. But is that just a document? It must be a package right? And then we have the “end user documentation” of FPM. Isn’t that just a “translation” of the formal spec?

Conclusion: we will allow language: Gujarati | ગુજરાતી syntax. First part must be language name as per ISO 639-1. But when allowing or disallowing we will use the entire value of language: key. So you can not have two translations with language: Gujarati | ગુજરાતી, but you can have one with language: Gujarati | ગુજરાતી and other with language: Gujarati | ગુજરાતી (simplified).

What if there were fonts that can be distributed as FPM package? They can chose to inline the .woff etc file in the package itself, but they can also link to CDN hosted font files (this is a bit privacy issue, we do not want Google etc to know what page is being viewed, and today Browsers re download all assets, I may be wrong, to avoid timing attacks for fingerprinting who you are or what websites you visit).

We can do the same to distribute other assets, say you want to share images with the world, you can upload it to unsplash or create a FPM package and put it on Github.

Soon people will start indexing such packages, and you will get a more direct reference, if anyone is using your assets you will be credited in the fpm.dependency of that package.

One question would be should these assets be copies over or given that each FPM package has a URL those assets are directly linked? Direct linking may not be good, as people will have to pay for their package getting popular. We should copy over.

Should we have a manifest file that tells what files to copy over? Should we have an asset folder which gets copied over? What about if multiple packages ended up taking the same name? We should copy the asset with the full package name (or package alias).

How would we refer to such asset?

We can make the URL resolve into /assets/foo/path/of/image.png. We can even do this on demand, the asset package may have many files, but we only copy over the ones referenced in current package.

What if this is transitive? If x is being built and it depends on a, which needs asset f1:foo.png. Should we create assets/a/foo.png or assets/f1/foo.png. The problem is same f1 could be imported by multiple dependencies with different names, so we end up having lots of copies of the same.

To solve this folder name must match full package name, not just the alias.

29th Dec 2021

Was reviewing the latest PR done by Arpita, and this thought occurred to me: we currently do, what Jest popularised as “snapshot testing”, we create “snapshots” of generated HTML files, and ensure they do not change.

I find HTML a bit noisy, reviewing change in generated HTML is not fun. Further there is a question of coverage - not all features of fpm could have been used in the .ftd file, maybe it is bare-bones, and when someone uses realistic .ftd files, maybe things would not go as expected.

Shobhit just implemented support for FPM/prelude.ftd, this file if found, is prepended to every .ftd file being processed (except FPM.ftd file).

Usually you would have a config.ftd file that can store a bunch of things that you want available everywhere. Also you would want to include one or more component libraries in your ftd files. Putting them all in FPM/prelude.ftd saves you from repeating the same lines in every document, making documents that bit cleaner.

Recently we discussed we will start publishing (copying) FPM.ftd in .build folder during build, and how we can sometime download just the FPM.ftd for an FPM package instead of downloading the entire package .zip file. It sounded like a great idea, and Arpita implemented it right away, and she did one better, she included some extra data also in the FPM.ftd file.

This opens up interesting possibilities, now we can include all sorts of generated package meta data and make it part of the published FPM.ftd file. We can show number of files or word count of the package, we can include LICENSE, even sitemap, latest snapshot versions etc etc. We can anticipate a lot of need, and most of the “metadata” needs of fpm can be satisfied by putting data in FPM.ftd.

So currently we also publish /FPM/index.html, which is generated out of FPM.ftd file in the package. With this idea, of making lot more data available to FPM.ftd that becomes even more interesting.

We will create FPM/info.ftd file, which will have all the FPM.ftd data available, and would be rendered in FPM/index.html, so we can have a proper package meta data page as well.

For any package, we have two potential source of truth, the repository where you can get the package source from, where it is presumably being maintained, say Github, or the actual built and deployed URL where things are hosted. Both contain FPM.ftd file.

Consider translation status. Say the original package is trying to create a status dashboard for all translation packages, showing like how many documents in that translation are upto date, missing etc. To compute this information fpm can download the latest version of each translation package, but this information is coming from where source repo, like Github, so zip may point to main branch’s .zip file.

If we trusted the main branch the problem is end users are viewing the translation status page, and they may see, oh this package is 100% translated in so and so language, but when the click on the link they may find that its not so, as there is a possibility that things have landed in main branch, but not yet built and published. This can happen because say deploy failed due to some reason.

So it seems that the <package-name>/FPM.ftd should be considered source of truth when its materially different from <package-repo>/FPM.ftd.

28th Dec 2021

Seems it takes less time for Arpita to build a feature than for me write what should be built! Just like that she implemented package with Golang like identifiers.

Consider this FPM.ftd file for instance:

Without searching now you can just copy paste the package id in browser and see the content of the package. So much better than the earlier approach of searching for the package in the package repo, and hoping the repo index page links to the published version somewhere.

Arpita pointed out that one of the cool side effect of this design (the fact that FPM.ftd is now downloadable, without having to download the entire FPM package) is that for many cases we will not have to download full package, eg when specifying translations, when a FPM package is specifying the translations, we only specify the package name, the actual language of the package is part of FPM.ftd meta-data of that package, and she used to have to download the entire FPM package zip file just to get the language (so she can show it in the translation dropdown).

So we are going to approach few open source projects to checkout FPM and our translation features soon. For us to setup the demo properly and get the discussion rolling, it would be good to import a the project we are targeting, convert them to a FPM package and publish on some temporary URL for demo purpose.

But we do not want Google to index our page. We can do robots.txt and block indexing, but its even better to add canonical URL so Google gives proper credit to original URL for improving their search ranking.

So we will support canonical-url key in fpm.package:

Shobhit implemented document title support yesterday. Till now all HTML files generated by FPM used to have title ftd, after this pull request the title is picked from content of FTD document being process right now.

In FTD we have a concept called region, any FTD element can use region to express their semantic meaning. The idea is to let screen readers, search engines etc discover what is the heading, or what is the role played by different elements, so they can skip over things and focus on main content etc.

We have regions like h0, h1 and so on for headings of different levels. To compute the title we look for the first text block with h0 region, if its not found we look at first h1 and so on. If no region is set then we look for first text element. If that is also missing then we assume the document-id (relative file path of the file being process with respect to package root, without the .ftd extension) is the title.

[Shobhit’s PR for dependency aliasing][pr] was merged today. This brings us one step close to changing a theme should be just one line change (its a bit harder than other systems where theming is well defined, you always know what you are theming, Wordpress support Wordpress themes, which only work for blogs, and say Winamp supports Winamp “Skins”, which only works Winamp, you cant use Winamp skin with your Wordpress blog for example[1], with FPM you all sorts of UI, music player to Browser, to pricing page to Resume etc can all be “themed”, and we want cross portability across all of them “as long as it makes sense”: follows the same “document interface”).

This is how you create an alias:

We have used as theme to create an alias theme for the package fifthtry.github.io/blog-theme.

In order to use it you can do this:

We are importing fifthtry.github.io/blog-theme by importing theme.

Note: We can also import the full package name if we wanted.

[1]: In case you are wondering why would anyone in their sane mind want to include a music player theme in their blog? Say you are discussing the UI of the player, say you want to show how the UI works, what would you do? Take a screenshot? And you call that action of a sane mind? What will happen if that theme changes? Are you going to come back and update the screenshot? That would be lunacy my friend. Just import the FPM package for the Winamp theme, pass the data that Winamp would have passed it (if we were living in the world where people are not, you know, insane – creating UI in platform after platform, even more insane where UI of one “framework” on same platform can’t use UI of another frame, I am looking at you JavaScript, with your framework pared – and using FPM for example), and now you blog post contains always refreshing, true, intractable Winamp screen. FPM makes it possible. Why would you not?

27th Dec 2021

Arpita just finished translation UI (and basic translation feature in general). I started writing update about translation feature on Journal, but I feel this deserves a video update. Let’s do a bit more polish, create different states, currently all translations are missing, do a bit of UI polish and towards the end of the day create the video.

In the meanwhile you can check it out here. Notice the links in the header, they take you to respective pages in other languages.

Super awesome job 🙌, Arpita!

26th Dec 2021

When using fpm build locally vs when using it for GitHub Pages hosted site there is a difference. Locally we build everything in .build folder directly, and URLs are /x/ for x.ftd file. But when hosting a site on Github Pages, without using custom domain, the URL GitHub Pages gives us is <account-name>.github.io/<repo-name>/x/. So every URL is prefixed by /<repo-name>/.

One idea that came to me was we use HTML’s base tag, set the base to either / on local or /repo-name/ in Github Action for GitHub Pages, and use relative URLs in ftd files. Have implemented this feature.

Other idea that came to me after implementing first solution: we create a folder <repo-name> in .build folder, so when browsing locally we get the same path as when browsing remotely, all URLs that we put in ftd files have /<repo-name>/ prefix.

Not sure the pros and cons of the two. Former is implemented already so that’s one pro. Later somehow feels cleaner to me. Not yet sure.

If you are using GitHub Pages, build your files with fpm build --base=/ locally and fpm build --base=/repo-name/ in your GitHub Actions, and use remove use x/ instead of /x/ when creating links in your ftd files.

fpm already supported macOS (we develop it on macOS) and Linux (our CI runs our test suite on Linux), but we never tried it on Windows so far.

Ganesh has been trying to use FPM on windows, and he had to install Visual Studio and Rust to build it. He had Windows 7, 32-bit based really old laptop, and turns out Visual Studio itself doesn’t support Windows 7/32-bit combination. He upgraded his laptop’s hard disc to SSD, and updated to Windows 11, and got Rust to compile FPM. We came across SQLite related issue, some SQLite DLL was required, so I switched to bundled SQLite, and it finally compiled.

But the fpm build did not work! It did not find any file. Turns out our code had hard coded / in a bunch of places, Shobhit found and fixed some of those issues, and after this fpm build is working for Ganesh.

Shobhit basically followed Microsoft Documentation about Rust setup and it just worked. Cool thing is the compiled exe works even on machines without Rust, of course, and Visual Studio (was not sure about this) installed.

We have not yet created any automation for creating exe every time a new release on FPM is created, we will figure out this soon (Jan end timeframe). For now if you need latest FPM exe, shout out to Shobhit on or discord channel and he will do it for you.

We have not yet run our test suite on Windows actually, only verified basic feature manually. We use fbt for testing, a testing framework for CLI applications, we have developed at FifthTry, but it too doesn’t support Windows yet.

Was looking at this thread on Reddit: Filename with dots or Unicode chars not working and I kind of agree with this sentiment:

I find it little disconcerting that a “modern” language has such an arcane restriction about Unicode filenames.

Reviewing the non-ascii-idents RFC that Steve Klabnik linked to, I kind of only agree with only of the drawback listed there:

Homoglyph attacks are possible.

Overall I believe we should go ahead with it. Rust has decided to despite the issues so should we.

What would it entail? What would we have to do? We should support unicode in FPM package names, as well as identified (component, variable, record names etc) in FTD. Any character is allowed in names? No, ID_Start and ID_Continue character classes are defined in Unicode, so all our “id”s should accept allow them (minis keywords reserved by FTD itself, eg record).

Other than Unicode stuff in package names, we actually have some restricts on package naming, so wanted to write down my thoughts about them. Currently all packages in FTM are distributed via Github. In short future ~ Jan-Feb 2022 timeframe we should also have fpm-repo that can be used for publishing packages.

fpm-repo is going to be self host-able, as well as SAAS version. But the distinction, if something is self hosted or on some package repository should not exposed to users of package, so authors should easily move their package from self-hosting to SAAS or back without breaking anything.

Moving from Github to fpm-repo should ideally also be seamless, but currently we expect the package to specify repo: github. We also have domain setting for each package, so domain name can be considered as part of a package, if we did that then the published version of package may include some information about how to fetch the source version of package, so in dependency it would be enough to give the package name, (with domain name as part of package name).

I was going in another direction, but I think I like this decision even more.

24th Dec 2021

After fixing font issue in FTD, finally we got FPM working with custom Google Fonts (any font for that matter should work).

We had an easy way to fix and a “correct” way to fix it. The problem is Font is not a simple file. Since we have a lot of languages, we can not embed symbols for all characters in all languages in a single font file, as that would become a large file and would slow down websites (and waste CPU/battery/bandwidth). Similarly many fonts design regular font face, italics, bold etc separately, previously people used to just algorithmically slant the characters or make the font bold by thickening the lines, but for best effect font designers create custom design for one or more/all characters for each font style. So we can not put regular, bold, italics etc in one file as well, as that too will be wasteful.

So CSS lets you specify multiple font files, for different unicode range and font face etc, and let browser download the font they need based on the content of the current page.

So coming back to problem: we only had place to store one font file, which clearly won’t cut it. The easy way to fix was suggested by Sourabh to allow people to pass URL of CSS file, eg Google Font recommends something like this:

@import, or the <link href="<some-url>" rel="stylesheet"> equivalent can be used. So we can ask people to give us CSS file URL as font file, and we can include it in the page. This does solve the problem, its easy. But it has a problem: it allows people to now provide arbitrary CSS file, there is no way to check the content of the target file to ensure it only contains @font-face declarations, it can contain any CSS, and it becomes a way for people to include CSS in FTD files.

This is a problem because FTD is designed with multiple target support, we currently compile FTD to HTML/CSS/JS, but in future we will compile to Swift code or Kotlin etc for mobile devices, or C code for ncurses based terminal UI support and so on. We are taking inspiration from CSS, the attributes for example, but that’s it, those attributes will have equivalent in Android/nCurses etc, or we will try to degrade gracefully.

If we allow CSS then suddenly our layout kernels can be misused to give arbitrary layouts, and suddenly FTD files won’t be compatible with our future FTD rendering targets. We believe CSS is over designed etc, and so is JS. We can create a browser from scratch without CSS/JS overhead, which is capable of rendering FTD files, so even for browsers we do not need the whole thing. We can maybe even use canvas API to use such custom rendering even for current HTML based browsers.

So we can not include CSS. What we did instead of looked at @font-face from CSS, and created an equivalent for the same:

We support all the attributes on fpm.font, with simplified names, eg name instead of font-family, or style instead of font-style etc, and removed src in favour of individual font types supported, eg woff or opentype etc.

So now we generate the CSS instead of getting the CSS from user, this was we know how to translate this transparently to Android when we come around to supporting native Android rendering and so on.

In future we may create a utility fpm css2font that will take the css URL from Google Font etc, and auto convert it to ftd.font specs that can be added to FPM.ftd file.

23rd Dec 2021

So I said including content from other package is proving to be challenging. Lets see the problem. We have to modes of including contents from other packages.

22nd Dec 2021

Since Monday, I and Shobhit has been spending lot of time discussing how best to allow people to include content from package into another.

Things are easier with data. Say you have a package that has defined some list or variable or some record, you can easily import that package, and iterate through that list or use the variable, and create your custom visualisations etc. This part of FTD/FPM is working well already.

But when it comes to text content, it proving to be surprisingly hard.

21st Dec 2021

When we are writing software we do not start from scratch, we take existing libraries and build on top of them. When writing on the other hand, say a book, we somehow live in the world where the assumption is every last word must be written or well its not a book.

It is not like we do not have open licensing available today for prose and creative writing. Creative Commons is out there. Wikipedia is successful project using Creative Commons. Github today has 147,082 Creative Commons repositories.

Even software documentation like Django documentation, is available in permissible license, even though it is not Creative Commons. Significant number of open source software ships with documentation, and usually that documentation is available in permissive licenses.

So we have huge amount of content available today. And yet we almost never see the content being “remixed”. Work being built on top of existing stuff. Everyone writing a new book on say Python feels they must start from scratch.

This I believe is bad for the world. We may think like we have a lot of books but we do not have enough books written. I once tried to write “Python for PHP Developers” and I got quite a good response on Reddit etc. We do not have enough intermediate level books.

But much worse than books, the real problem is ironically still too many books. Like say you are an intern and you want to join FifthTry, you will have to learn Rust, but then also Diesel, and PostgreSQL. And yes Django. And possibly even Elm. Some JavaScript. Some CSS. And of course enough about Linux, and AWS? And this is assuming they do not have to learn any FifthTry specific stuff. Which they still have to. What is FTD? How we deploy our services, how we do things and so forth.

Imagine there was a great book written about each of these topics, am I supposed to hand that many books to an intern who joins us? Maybe not all of them but it still many books.

How about if there was just one book? Think about it. Do we really need to learn whole of Django? Of course not, we only use Django for database modelling and migrations, so they must not actually read about django views and admin and template and so on. If you think about it telling an intern to learn any of them is itself really hard. How much of Rust should they know? How do you even talk about it? Every team that uses Rust uses a different aspect of Rust.

Should you learn about things that’s not directly pertinent to your job, the exact task? Of course learning is never bad, but we have only so much time, and if you not smart about it you may end up spending too much of your precious learning time learning things that may not be used for the job at the expense of other stuff that you will need. Knowledge acquisition simply can not be done in depth first fashion, since it’s a bottom less pit.

In summary we want people to be able to create books or dedicated websites for learning something, or compilation of information, knowledge, data etc, by using other packages as dependencies. This can allow “knowledge curators” to create learning paths for specific audiences.

20th Dec 2021

Me and Arpita worked further on the design of translation UI. Check out this commit to see the basic skeleton of our design.

The key idea is now we will have multiple base HTML files, with-fallback.html and with-message.html, which will be created with content of more than one FTD documents. For example with-message.html will render a “message document” and a “main document”. Further, individual documents can be shown and hidden using basic DOM style manipulation (display: none etc).

We are going to need a new feature in FTD, “message-host”, which will allow an event handler to message the “ftd host” (also in future we are going to rename FTD “Library” terminology to “FTD Host”).

Our with-fallback.html will come with a show_main() and show_fallback() host messages that our “message ftd” file will be able to trigger using $on-click$: message-host show-main etc.

We are quite happy with this design for now. I wrote couple of feature requests that can be implemented using this: “banner support” and encryption support.

19th Dec 2021

A relatively strong consideration for designing translation, especially translation of versioned packages is that “translation can not alter logical content” rule. Consider 1.0/intro.txt vs 1.1/intro.txt. Consider a question: who decides if 1.1/intro.txt must exist? Original package only, or can translation package also decide this? Is it possible that in original we only have 1.0/intro.txt and no 1.1/intro.txt, which means as per the maintainers of original package, who supposedly are working closest with the actual software being translated, as per them there is no need for 1.1/intro.txt, that there is no software change between 1.0 and 1.1 such that intro.txt content changes. But can translation maintainers take overwrite this call and still decide to have 1.1/intro.txt?

What if the original 1.0/intro.txt had only so much content and translation team not only translated but decided to expand the intro and cover more topics in the intro? If they do then original and translated intro have logically diverged. It would then not be accurate to say that this is just a translation, it is not an independent work which was derived from the original.

I am not saying logical changes, derivative work shouldn’t be allowed, just that one should be aware when they are doing it. Some authors may be okay with their work being translated “faithfully” in other languages, but may not like their work editorialised or “improved upon” by others (and sold / marketed in the original author’s name). I as an author may not be able to read how you have altered my work in Chinese or some, so how can I as an author endorse it? I don’t even know what’s in it, maybe the change is minor but its really hard to say what is minor and what is not. A little thing the translator decided to omit as its too early to mention this feature, maybe the reason people would not buy that product because the author knew that that feature must be mentioned earlier as author knows the audience better than the translator.

Bottom line is there should be a clean line between translation and derivative work. And authors should be allowed to say they are okay with strict translation and may not have energy to whet the “editorialisations” and may not want derivative work.

So when we are talking about translation so far, we are talking about the stricter translation, no logical changes, as faithful as vagaries of natural languages allows one to be.

Let’s talk about tracking now. Tracking basically says “hey this document has changed, and you said that document depends on this document, so maybe consider updating that document”.

Now one can argue that tracking should not be allowed in translation packages. If you want to be strict, you maybe tempted to say the only document a translated document should be allowed to track is the corresponding document in original package.

By and large I agree with assessment. There is one exception tho: what if say you are also maintaining a “translated glossary”. This document may not exist in original, but in every translation one must maintain a document that lists what’s the translated term we are using for each interesting technical term in the original language. If we start from scratch, two different translators can pick two different ways to translate a term, and that may not be good for readers. So this document should be present, and one can argue every document in the translated package should be tracking the glossary file.

We are planning to implement a feature in ftd, which can make this consideration redundant though, do not put translated term in respective documents at all, but refer to them from a common glossary file via ftd markup references so if translation for a term changes all documents using that automatically get the latest version.

Where does it leave us? Disallow tracking in translation packages?

So Arpita has heroically built the entire translation feature almost. fpm translation-status is working. It looks for status of all documents in original package by reviewing content of .track and shows status for each file like “Never Marked”, “Update Date” etc.

TODO: Arpita to give me sample output of the command.

She has also implement fallback based build: if django-hi/intro.ftd doesn’t exist we look for django/intro.ftd.

Next: We will build fpm mark-upto-date to keep updating translation status.

We also discussed if django has N dependencies do we auto include them in django-hi if django-hi is a translation? Or should each translation mention each dependency explicitly? For translation we may want to use the translation of a dependency instead of the original package, and while doing so we would need to be able to rename and keep using the same name.

Next we also have to add language in fpm::Package, and ensure that key is set if current package is translation-of some “original package”. Original package must also have the language set and the languages must be different. Also translation-of can’t be “chained”, if this package is a translation of some other package then no other package can translate this package, everyone must translate the same package, each translation “cluster” has a single original translation package that every package in that cluster is translating.

Finally we need to come to UI. We have a few scenarios, say translation doesn’t exist for some document, in which case we have to show a message on top saying you are seeing the English version because this document is not yet translated in Hindi. Similarly we need UI for “never marked”, “out-of-date” and “upto date”.

We want people to be able to control these messages, these messages themselves must be translated. We will have files FPM/{missing, never-marked, out-of-date}.ftd, if they are not present we will have some fallback message. When creating a new translation using a future fpm create-translation command we will auto create these files maybe.

In never-marked and out-of-date scenarios we will have three FTD files to build, which will go in the generated .html file. The “message” file will show on the top of the page, and then one of the two, original and translation, would be shown and the “message” file will have a button with an $on-click$: call-js <function-name>, and we will have FPM.js file, which will have functions show-original and show-translation (and a “client variable: fpm.showing-original).

We will also need other server variables to be used when building “message” file with variables like original-language, translation-language, translation-status: "never marked" etc, original-marked-on-date, original-latest-date, original-diff etc.

15th Dec 2021

When referring to a document, we use a document id. The document id for a document in current package is the relative path from package root. If we are referring to a document from another package, we use <package-name>:<document-name>, the colon separated path.

On its own if you see x, it can either be a document in current package, you may ask why no x.ftd? Are we allowing x/index.ftd to be referred to by x? No. Our index.ftd is only special at package root level, and not at individual folder level, we follow x.ftd and x/y.ftd convention and do not fall back to x/index.ftd, x/y.ftd as say node etc do. There are advantages to fall back, eg whole of /x/* is in a folder x, this leads to some simplicity, eg if you want to ignore that path, you have to no ignore x/ folder as well as x.ftd file.

We also have complications around import path in ftd documents vs file paths on command line. If you want to import x.ftd you write -- import: x, without the extension. Does that mean when you want to refer to x.ftd anywhere in FPM context it must be x? Like on command line if you want to add tracking on x.ftd or sync x.ftd do you use x or x.ftd?

We can have a rule, if only ftd file is meaningful in any context, eg in import statement, you can not import a .txt file, but you can track .txt file, so if only .ftd is allowed for something we use the path without .ftd extension and any file can be passed we use the extension.

We can also standardise on file vs document terminology. Document in FPM context strictly means it is a FTD file, where as file means any file. So function names, parameters etc can use print_file() or print_document() etc naming convention.

There is one more complexity. Say you have a logo.png file, there can also be a logo.png.ftd file which contains the meta data or web index of the logo.png. We can argue that this is too much complexity and lets have no such special convention. For example this causes another complexity as to what should be the URL of such logo.png.ftd file? The URL of logo.png is /logo.png if it is at package root, but for x.ftd the URL is /x/ so with this logic the URL of logo.png.ftd would be /logo.png/, and having different content based on presence or absence of a trailing / in URL seems to be a bad idea. Or maybe we do it based on “accept” header, if you request an HTML for /logo.png you get the content of logo.png.ftd and if you request image you get the .png. But then how would you link to the file? Say you want to open an image in a new tab?

14th Dec 2021

fpm cli currently is littered with println() and eprintln() calls all over the place. This works okay if we assume that output will always be on terminal, but we want to support JSON output as well. Also even if we did not want to, in general this is bad software design as if the code on how to show output is spread over the entire crate then its harder to modify it, test it etc.

What we should do instead if each of the commands that we have must have a corresponding output type define. Currently we return just a success indicator, and the return is returned as some custom error type, which in general sounds good but has no provision for warnings etc, say if you are building cp someone asked you to copy 10 files and you failed to copy the last file, should you only get the Err part, then you will have to stuff information about the files that were successfully parsed inside Err, or you say if even one file was successful then we return Ok.

This is just mis-using Result. We can use Result::Err to indicate something really bad has gone wrong, we could not proceed at all, say the command you tried to execute does not exist, or the config file is not found. But all other wrong, eg a given FTD file has bad syntax or could not be read due to permission issues, we should send all as Result::Ok.

Further we should have a type for each command which represents the output, which can be printed out on stdout with colour highlighting or converted to JSON etc. Also, main() should do the print to stdout or convert to json etc and not individual command handler functions.

13th Dec 2021

• fpm sync <list of files file>
• fpm status <file> to show all statuses for the file
• fpm status to show status of adhoc tracking but not translation stuff
• fpm stop-tracking
• fpm diff <file> to show all diffs (including tracking diff, translation diff)
• fpm diff -a to show everything for every file`
• support tracking, sync, diff for any file, not just ftd files
• allow any file type to track any file type
• fpm translation-status
• fpm translation-status <file or folder>
• fpm mark-upto-date a --target b --on <timestamp>
• fpm diff <optional-folder> to only show regular diff (only changes from .history file)

12th Dec 2021

Been looking at code and found a struct fpm::document::Document. Not too happy with its design right now.

The idea is we have a lot of commands, and many of these commands go through all the documents in the current package, and fpm::document::Document and the corresponding get_documents() gives you a vector of these documents for easier time handling all files. The main job this get_documents() is abstracting over the directory walking so you do not have to worry about anything and you can just do a for d in get_documents(). Another thing get_documents() takes care of is ignoring documents that should not be handled, eg .history, .track, .build etc special folders.

I was thinking about what’s bothering so much about it. The first thought is abstraction and frameworks. I kind of am not fond of them, unless there is a strong reason. It like a trade, you want to have a nice Mercedes, sure, go ahead, you just have to pay for it. You cant just wish it. If you want to create a framework you have to understand all future needs. You have to educate people, it must be well documented. There must be abstraction boundaries. Everyone must understand what this framework is doing etc etc. People look at framework and go say I will also make one. But I will do all the work later. If you wonder where tech debt comes from, this is where it comes from. You went to a Mercedes showroom and drove off with a shiny car, and told them you will pay for it later. Live with your “bailgadi” instead.

10th Dec 2021

We already supported the conversion of markdown files to HTML during the build process. Shobhit has further added “markdown index” support, any directory that says a with README.md in it, without a corresponding .ftd file, we render the content of the README.md at /a/.

Vipin needed font support, realised we have not released anything for a while so created a new release.

Was discussing package interfaces, and we have to start creating some of these package interfaces now, now that package and basic machinery is working fine.

The motivation for package interface, and aliases, is to make it easy to change themes and “component libraries”, without doing much change in your package.

When using a theme or a component library you have to first add them as a dependency in your FPM.ftd:

If we just do this, we can now import some-lib from any file in the foo package.

But what if we want to change that some-lib? First of all, we someone has to ensure that the new library is compatible with some-lib, else switching would be a tedious process, we will have to update all calls of some-lib with a compatible component or type in the new-lib. What guarantee is there that new-lib will have corresponding types and components (with same arguments, type and semantics?).

Say some-lib has exposed a component heading for authoring headings in documents:

9th Dec 2021

• fpm sync <list of files file>
• fpm status <file> to show all statuses for the file
• fpm status to show status of adhoc tracking but not translation stuff
• fpm translation-status
• fpm translation-status <file or folder>
• fpm stop-tracking
• fpm mark-upto-date a --target b --on <timestamp>
• fpm diff <optional-folder> to only show regular diff (only changes from .history file)
• fpm diff <file> to show all diffs (including tracking diff, translation diff)
• fpm diff -a to show everything for every file`

We won’t delete the history of a file when a file is deleted. We currently have .history/foo.<timestamp>.ftd for every modification of foo.ftd. We will soon also have .history/foo.<timestamp>.ftd.info.

In some cases only the .info file may exist, when foo.ftd has been deleted.

.info file will also contain the “commit message” of a change. And in future when we implement history browsing it will also contain a flag that tells the UI to ignore this particular change saying it’s a minor thing and is distracting. We don’t actually delete the timestamp file and merge because timestamps are created on sync and after sync it gets published, and others may have stored that timestamp in their .track file.

Technically, we can manage to miss .timestamp files from a .track file by using the immediate previous entry, but it will cause a double review in edge cases. I prefer we never delete the history.

One of the issues with SQLite is we may not be able to do an optimisation.

When things are rendered via fpm-repo things are fine, but when using fpm build, things can become a problem if the package size or complexity grows. With support for HTTP request handling and SQLite queries, and we will be adding a lot of stuff, we are just getting started, the time it takes fpm build may start becoming an issue.

One way to speed things could be to do build caching, do not rebuild something that has no change. Consider the simple case, a.ftd, which does not import any other document. Consider also we do not have dynamic features like fpm.now which returns the current time. In such cases, we can choose to not rebuild a.ftd if the content of a.ftd hasn’t changed.

We can detect if a document uses another document or dynamic features, and we can store this information, and use it when to decide if we should rebuild things or not. Eg, if we know some document has used fpm.now and has no other dependency and that document’s content hasn’t changed, we still want to regenerate the corresponding html, as fpm.now returns a different value on every run.

We can do this optimisation for sqlite as well if a random sqlite file has been used. If the sqlite database file has changed then we need to rebuild the document that queries that sqlite database even if the document itself hasn’t changed.

But when we talk about query against package database, a sqlite database that we would be creating/updating based on the content of all documents in an fpm package, that sqlite will always change, and any document that queries that package database can never be cached.

Unless we break the package database into smaller databases, say for each record we have a sqlite file, and some document is only interested in instances of that record, and we know that only a few files have created an instance of that record, now the out of date graph would be a lot smaller, only those few files.

Smaller files have problems tho, some queries would want to use join and if the tables in different database join may not always be possible (not sure how things would be after the support of referring multiple sqlite databases for a single query, can it negate this drawback?). Not sure how foreign key would work etc.

One thing is there that we can do both, generate big databases and individual databases as well, and do it opportunistically, only generate the databases that were referred by any of the documents. This way we can even choose to not generate the big package database if no document is querying that database.

How about a database for use for other packages? Say package a wants to query database of package b? That is slightly simpler because then you are not worrying about build time but sync or publish time. It may be okay to do more work during publishing.

But then again why are we calling build all the time? Why are we worried about performance of the build at all? Because we currently do not support auto reload?

Auto reload means we have to support the demand generation of pages, in which case we don’t have to worry about caching, the moment we introduce support for fpm server, even without auto reload feature, this whole discussion would go moot.

Making fpm build fast would be important, but then we can ignore it as it’s no longer urgent, as so far my concern was author edit workflow, which would be improve much more by auto reload and fpm serve than making fpm build fast.

Till now fpm build used to only process .ftd files in an fpm package. Shobhit has added support for handling static files (files other than ftd and markdown, and not ignored by .gitignore/fpm.ignore etc).

Now we can add images etc as part of an FPM package and they get included in generated static site.

Since we kind of now include all files in the current directory, it may be a good idea to review the files that are being processed and ignore files you do not want published, both for performance and security reasons.

One kind of thing we have to be constantly aware of when writing/reviewing any FPM code is accidentally copying say /etc/passwd to the built file. This can happen say if we started support symlinks, and someone has a package that you are contributing to, and symlink gets you to accidentally copy more stuff to .build and publish to the world.

It can happen more than by just symlink, say in $processor$: package-query, if the value of db is set to say ../firefox-password-store.sqlite file, again we may accidentally read beyond what we should, which is the content of the current package or any package that is a direct dependency of this package. In fact, if current package depends and a package a and a depends on b, then we must not allow current package files to directly reference files in b without adding an explicit dependency on b, as while this is technically not a security issue, it causes issues if a removes the dependency, the package will suddenly stop building.

We are shipping a binary that runs on people’s machines, this is a threat vector, and while its easy to audit code for direct threats (attempt to do mining, or scanning stuff), we are also letting our binary download packages from the internet authored by others, and we have to give guarantees that those files can not do any harm either.

Our FTD must not get any feature, for instance, letting people directly execute commands from a package, without user understanding the command (this can happen when a package is being downloaded for the first time, we should show all the external commands this fpm package is going to be able to run etc).

The thing is this is very tricky, we can not say we don’t want to let package authors run anything, that will significantly decrease the power of FPM packages. We have to allow people to query data from their trusted data stores, to make it part of generated documents, if they want to. But we can not surprise people. Even HTTP requests are sensitive, not just command execution.

• fpm.markdown-filename etc
• fpm.package.name vs FPM.package-specific-variable
• import path disambiguation, can be use : as separator
• import: <package>: is ugly, import foo means foo is either in current package or a package which is a direct dependency of current package

1. a (local), a: (package index), a:b/c (-- import: a: is ugly)
2. a (local or package-index), a:b/c (pretty)
3. @a (local), a, a:b/c (explicit)
4. a (local), @a, b/c@a

Currently ftd $processor$ can only be used for variables and lists, we have to extend it to use on component construction as well.

We may also want to specify more than just the body as well, eg other headers.

Arpita implemented these two commands.

We had a funky fpm a tracks b idea, which no command line tool does. She used the fpm tracks destination source syntax. We should make it maybe: fpm start-tracking <f1> --target <f2>. Also the code assumes we will only track ftd files, but we have to be able to track any file. Remove .ftd prefix handling.

fpm mark-upto-date <f1> --target <f2>. Here target would be optional eventually, if the target was missing and f1 tracks more than one file then the cli would prompt which file.

If the file is not found it silently returns, should report the error. If file is already being tracked it should show some message.

Not sure if fpm diff should show all the diffs, local modifications and tracking diff, or should we have a separate command.

8th Dec 2021

Recorded a design review of what I am calling the upcoming “FPM Web Framework”.

7th Dec 2021

Yesterday I started implementing SQLite support for FTD documents via $processor$ mechanism of ftd. It was stuck on some missing helper utilities in FTD, you can see the two functions at the end with todo!() as their body.

Today Arpita is feeling a bit better from the cold, and she has created PR implementing those functions in ftd. She can’t not work even when down it seems!

On that change landed I got the $processor$: package-query working. Update the documentation as well.

During implementation, I found out that serde-json support for rusqlite is actually, umm, not sure what word to use, not buggy per se, but kind of not as useful as I first thought it would be. It converts only the text field to serde_json::Value, when other fields like integer, float etc can also be easily converted to serde_json::Value. I implemented a simple converter for other sqlite data types as well.

We are not done yet. We need the ability to pass parameters to queries. Also, those parameters should be able to refer to other ftd variables. And finally, we should be able to read into more types, currently, we only read into records and a list of records.

So we have reasonable sync/status/diff working now. We are going to soon implement “ad hoc tracking”, basically a .track folder that contains foo.track like file for each foo.ftd file. status/diff will look into the .track file for any file and see if they are up to date with respect to the files being tracked.

Once we have ad-hoc tracking, we are going to work on translation tracking, where a package would be able to designate itself to be a translation of another package using translation-of key in FPM.ftd. When a package tracks another package like this it means two things.

6th Dec 2021

Jay has created a logo that we kind of like, and some initial design for our homepage:

Since we are anyways going to implement some sqlite support for package database, we may as well let people include arbitrary sqlite files in the package, and let people do arbitrary SQL queries on top of any of the sqlite databases in the package or any package that is a dependency of this package.

Change Request.

Partially implemented $processor$: package-query. Need two methods in ftd crate, waiting for Arpita to get better and implement them.

Recently I implemented $processor$: http, which paves the way for almost fully fledged “dynamic”/data drive documents. To really expose this feature, we can implement arbitrary URL handing, currently only the URLs corresponding to actual FTD files is available, this is too limiting as we can not have author page for example, where we need a URL for each author, and author list maybe in a table, or spread all over ftd documents.