Time tracker

THIS IS A WORK IN PROGRESS <

Below you will find log entries that describe the design/thinking process.

Idea <

The goal is to build a device that could allow me to comfortably measure time spent on a project. I want to have a full history of the time spent on particular project. The time log needs to be plain text, easily queried and human readable.

Process <

2023-08-23

And here we go with another attempt at prototyping this device. 3D printing is great, as long as it works. Any 3D printer issues can derail the entire process - I’m not here to practice my 3D printing troubleshooting skills. Another thing about 3D prints is that they’re not a good base to further refine the printed prototype. It’s a plastic shell and there’s not much you can do to modify its shape, without redesigning and reprinting. There’s definitely way more to prototyping than 3D printing.

I’m always interested in improving my prototyping skills and learning new techniques is definitely worth the effort. I have decided to try making something by substracting material, not adding it, like in 3D printing. I do not own a CNC router but I’ve decided to do my best with Dremel’s multitool and their Plunge Router Attachement. It’s a manual tool so, without a very disciplined process, the result won’t be perfect.

Below you can see my first attempt at cutting into HDF. The result is a 1.2mm thick device, with 4 buttons.

I’ve actually started by using MDF, but it’s slightly more difficult to cut and it releases an unpleasant smell, when routed. I have moved to hardboard, which lacks these drawbacks. My main gripe with it is that it’s difficult to find anything else than 3mm sheets. In fact, this prototype , is made out of four, 3mm layers of HDF.

I’m pretty happy with how the keys turned out. They’re just a part of the same HDF sheet, as the top cover. I’ve removed some material, from where the keys transition into the rest of that sheet, which allows them to flex, down (good) and up (bad).

The ESP32 board, Lolin D32 Pro, isn’t the best fit for this project. It’s too big and carries ports that are completely unnecessary for this project. I have ordered a better fit - D1-Mini32 board. It’ll be easier to route a slot for this board, since its size is at least 50% smaller than Lolin’s.

This entire experiment made me want to buy a desktop CNC router. Sain Smart looks like a good quality and price ratio option. Their CNC machines are even able of machining aluminium, but I’m not seeing myself going that far in the nearest future.

That was a great experiment. My toolkit grew and anything I make in the future can benefit from this experience.

2023-01-15

I’ve been using the Raspberry Pi as an input device for this project for a while now. The current database stores around 200 records. At the moment, I mainly use it to record my drawing practice sessions. I rarely prioritise this project which means I didn’t progress much on replacing Raspberry Pi with something more appropriate.

I have experimented with the common tact switches. Those usually require quite a bit of force to trigger. You can buy them with a lower force threshold but I used the ones I already had. The idea was to offset the spot where the user puts the finger, from the spot where the switch is located. That creates a lever, effectively increasing the force applied to the switch. I have 3D printed a small “keyboard”, into which I have inserted the tact switches. I was quite happy with the result. I could move the switches closer to the actuation point to tune how hard I needed to press the key.

In the meantime, I have found Omron’s D2LS-21 switches. Those come in two variants. Blue switches require 61gf, while white require 122gf, of triggering force. I have ordered both to test them. These switches give the same feeling as the mouse buttons do. Initially I felt the blue switch might be too weak, but it actually feels better than the white one.

Here is a quick prototype based on those switches.

While I’ll iterate further on the actual form of this device, it’s clear that without the status LEDs this device will be subpar to the Raspberry Pi device, I’m using currently. It’s necessary to be able to tell if a task is active or not.

That puts me dangerously close to a point where I’m thinking about designing a PCB, for the buttons and the LEDs. When it comes to building hardware, you have to be able to judge if you know enough to take the next step. Since I’m still unsure which surface, top or front, should host the LEDs, I probably shouldn’t rush into PCB design stage. I also shouldn’t make the prototype too complex, where the LEDs need a unique PCB. At least not without a good design of the form, which would justify the added complexity.

2022-08-19

This project’s frontend was never using any explicit JavaScript. I’m not a web developer and the entire ecosystem is foreign and seems unnecessary complicated. I have some experience writing JavaScript but it was always small, simple, from scratch scripts. I’m never getting on the npm bus. I’ll always be very careful with dependencies in all my projects. At the same time I, just like a typical technology consumer, expect certain level of flexibility from the tools I use.

Enter HTMX - a JavaScript library, which extends HTML. I’m not going to waste your time explaining how it works. Let me just give you a example of how it impacts this project.

Before, whenever I opened the list of the recorded time stretches (entered the URL in the browser), the server generated graphical representation of those time stretches and sent the contents of that page, with those images. This isn’t a problem when you have 50 time records, but it obviously doesn’t scale. Imagine you have 1000 records and any time you refresh the website it generates those images and sends everything to you - horrible waste of resources on so many levels.

What would be the expected behaviour? You open the webpage, which loads the basic HTML + CSS immediately, in the meantime the server renders the images and as soon as it’s done it sends that data to your browser, which renders those images in the correct location of your website’s layout. This way the image generation and fetching became decoupled from the actual HTML + CSS transfer. If the time it takes for the server to generate those images grows, it’ll only impact that second transfer. That doesn’t give me a permission to bloat the image generation though and I have to be careful to not make the user wait for those images for too long. Let’s talk about why that shouldn’t be an issue.

Previously I mentioned having to generate images for 1000 records and display those. That’s also not part of the expected behaviour. I don’t think anyone wants to see their webpage to be flooded with 1000 images. The frontend needs to provide a way to page those records in a reasonable way. Here, I imagine having a ‘Load More’ button, which asks server for more data.

This is where some JavaScript, in form of HTMX comes in. It allows you to attach server requests, which can use any of the HTTP method, together with a trigger, to any HTML element. If I then want to load bunch of images when the webpage’s HTML loads, I can use the hx-get="/records_uri" attribute, with a hx-trigger="load" trigger. If I want to implement the ‘Load More’ button, I can just use the hx-get="/records_uri_but_more" attribute on a <button> element.

Currently I’ve implemented asynchronous images loading. I still have to tackle the ‘Load More’ functionality. One thing I never knew is how people generate the URI for the REST API. A ‘Load More’ button needs to have a correct URI in the hx-get="/records_uri_but_more". If you’ve pressed that button once already the URI should point to another URI or you can call the same URI every time you press that button, but in that case the server needs to store a state to know what to server to the client, when the client request “more” for the 5th time.

I was lucky to stumble on the How did REST come to mean the opposite of REST article. I can’t say I understood it reading it the first time but I’ll once again reiterate that the web development world is a foreign planet to me. Adding the simple features using HTMX, together with the aforementioned question, helped me understand what’s this article about.

While I still don’t fully know how a proper URI for the third press of the ‘Load More’ button should look like, I know that the ‘Load More’ button’s hx-get attribute will have the URI that the client received after clicking that button the second time.

The article explains that in a proper REST API, the server should respond with HTML. Since it’s HTML the browser will just replace the current HTML with the new content. Part of that new content will be a new ‘Load More’ button, with an attribute which URI points to the next set of records.

This is one of the fundamental properties of the original idea behind REST API. You don’t need the documentation of the API. You just need the entry point. From there, the server will serve new elements that point to different endpoints. You can change what the server serves and the client rarely, if ever, has to worry about those changes. It’s role is just to present that content. I find this quite elegant.

One last thing I worked on is the new images for recorded time stretches. I’m thinking it’s better to use weeks as a grouping method. I’m also going for a more tightly packed layout. If the user needs to inspect a day in more detail then they can clock on one of the donuts, to get a more detailed view.

This concept isn’t easy to implement though. Currently I’m working on generating those donuts as SVG images. That’s a quite tricky task.

2022-07-02

I have modified the Postgres’ schema. I’m using ULID for projects’ IDs. Currently I generate those manually, using ulid-py. In the future I’d like to be able to create new projects using frontend’s interface.

This change, obviously, impacts how I insert the data into the database. For a while my backend server wasn’t updated to work with this new schema. This killed the entire system, for the time it took me to pull myself together and do the necessary work on the backend server.

Since the downtime was quite long, I managed to forget that this project is broken. That inspired me to tackle something I meant to explore for a while now - notification system!

Currently it’s difficult to tell if the backend failed to insert data into the database. The typical way to do it was to check if the new entry popped up in the frontend’s interface. If it didn’t I’d have to SSH into my VPS and check the Docker logs. I’m not that interested in checking the frontend’s log every time I finish a task and end time tracking. I also want to be certain that the new entry was actually saved in the database. At this point the goal was to implement something that would notify me if the backend server failed to insert the data in the database.

Simple, small goal calls for a simple, small solution. Something I can code in a single day. Around the same time I have read that Telegram’s API is quite simple and sending a message to a Telegram bot is very easy. I actually have some experience working with Telegram bots. I did use one before. I have used a Python module and it actually took me a while to implement the things I needed. After reading on how easy it is to send a message to Telegram bot using curl I have decided to not use any libraries to do this and write everything by myself. I really don’t need much here.

If you’re curious on how to do it yourself, here’s a quick tutorial.

First create a bot using BotFather bot. When you get through all the steps you’ll get your bot’s token - you’re half way there to be able to send a Telegram message through their HTTP API.

Using your bot’s token you need to get the chat’s ID. The single line below can do this for you. Notice I’m using pass command here. This tool allows you to store your passwords, encrypted with your GPG key, in your local filesystem. I saved my bot’s token using pass and now I can easily and safely read it back. You should be able to easily modify the lines below if you’d like to avoid using pass. The same goes for using jq - convenience tool which makes it easier to find the chat’s ID.

# get chat ID
TOKEN=$(pass tokens/telegram/my_bot)
curl https://api.telegram.org/bot${TOKEN}/getUpdates | jq .result[].message.chat

Expect output to be something like so (you’re data obviously will be different, especially id, which we’re here to get):

{
  "id": 111111111,
  "first_name": "Name",
  "last_name": "Surname",
  "username": "username",
  "type": "private"
}
{
  "id": 111111111,
  "first_name": "Name",
  "last_name": "Surname",
  "username": "username",
  "type": "private"
}

Now that you have the chat ID you’re ready to send a message to your bot.

# send message. 'chat_id' taken from previous step
TOKEN=$(pass tokens/telegram/my_bot)
curl -X POST \
  -H 'Content-Type: application/json' \
  -d '{"chat_id": "111111111", "text": "hello Mr. Bot, hope you are well <3", "disable_notification": true}' \
  https://api.telegram.org/bot${TOKEN}/sendMessage

The message should pop up in your bot’s conversation.

Knowing all this, it’s trivial to implement it in the programming language of your choice. My backend server is written in Nim, so that’s the implementation I went with.

import std/httpclient
import std/json
import strformat

const BOT_TOKEN = "imagine the token is actually here"

proc notify*(message: string) =
  assert BOT_TOKEN.len != 0

  let client = newHttpClient()
  client.headers = newHttpHeaders({"Content-Type": "application/json"})

  let body = %*{
    "chat_id": "",
    "text": message,
    "disable_notification": true
  }

  let response = client.request(fmt"https://api.telegram.org/bot{BOT_TOKEN}/sendMessage",
                                httpMethod = HttpPost,
                                body = $body)
  echo response.status

when isMainModule:
  notify("running notifier as a main module...")

Finally I can use it in my backend’s code like so:

    # Try inserting the data.
    if true == db.tryExec(query,
      rec.device_id,
      rec.tp_sync,
      rec.tp_start,
      rec.tp_end,
      rec.t_delta_m,
      rec.t_delta_s,
      rec.project,
      $(rec.client_ip),
      rec.comment,
      1
    ):
      # Succesfully inserted the data in the database.
      notify(fmt"You ({rec.client_ip}:{rec.device_id}) worked on {rec.comment} for {rec.t_delta_m}min {rec.t_delta_s}s")
    else:
      # Failed to insert the data in the database.
      notify(fmt"Failed to insert ({rec.client_ip}:{rec.device_id}) {rec.comment} {rec.t_delta_m}min {rec.t_delta_s}s")
      logger.log(lvlWarn, "Failed to insert data into DB!")

This is my ‘good enough’ notification system.

I also managed to improve parts of the frontend. The main goal is to be able to manage existing projects, create new ones and map them onto Raspberry Pi’s keyboard.

2021-09-12

I’ve worked quite a lot on, what many would call, the backend. Since my experience comes mostly from embedded programming, a lot of the concepts around backend infrastructure are new to me. Fortunately I know Docker well enough to be able to spin up a PostgreSQL database on a VPS.

At this point I already had a server, written in Nim, which received the data from the Raspberry Pi or the ESP32 prototype. What I had to add was the Postgres functionality. Fortunately Nim’s standard library already has a module for Postgres. After several days of trying I was able to write Dockerfiles which initialized and started the database and the Nim server. The database schema is still work in progress but it’s very much useable right now.

The next step was to write a web app which would read the data from the database and provide a webpage interface. Once again I chose to write the code using Nim and a community made web framework Jester. The current web interface is pretty simple but it does present useful information.

It’s obvious that the red time segments aren’t that useful just yet. The user can’t tell which project those correspond to and what time did those start exactly. It’s worth pointing out that those SVGs are generated by the server, when it fetches the data from the Postgres database. It might be obvious that its generated (because how else could it work?) but doing that, using code, is very rewarding

At this point in time I can: start time tracking on the Raspberry Pi prototype, recorded stretch of time gets sent to a server which saves that data in the database (with additional information: project ID, user ID and IP, etc.), the web application reads that data, generates the website’s content and serves it as a, decently, readable interface.

2021-06-12

I’ve tested the pHat Touch from Pimoroni, attached to a Raspberry Pi. It’s a good prototype of the functionality. It allows for tracking time of 4 projects, thanks to a script written in Python. Again, Python proves to be a great tool for very quick prototyping.

I’ve been thinking a lot about the design of the device. Ideas that I keep coming back to are: keys made from a thin, metal rods, wooden or ceramic (yes, ceramic!) enclosure. Those ideas pose manufacturing challenges, that currently I have no idea how to solve.

2021-02-27

After some 3D printing problems I’ve decided to take a step back. I’ve soldered a very simple prototype on a universal PCB. I was trying to take too big of a step. It’s difficult to juggle enclosure design, electronics design and software. Priority should always be given to the parts that matter most to the final goal. Beautiful plane won’t take you anywhere without a functional engine.

Making this prototype allowed me to improve the ESP32 firmware. It handles multiple buttons now and a simple debouncing mechanism has been added. I’ve also added various other improvements. The server code was also significantly improved.

I also realized that a visual feedback is very important. If I press a button, a corresponding LED needs to signal that the ESP32 is logging time for the specific project. I knew it’s important before I built the prototype. After, I’d consider it to be crucial. That forces me to explore the possible pinout configuration. I might learn that squeezing in 5 buttons and 5 LEDs might be difficult.

Building this prototype proved to be very beneficial. It pushed the entire project several steps forward.

2021-01-23

Recent experience with ESP32, working on the ESPotify project, taught me a lot. I feel confident moving in ESP’s ecosystem. This, plus need for a break from other projects, made me start working on the firmware for this project.

During two days I was able to write a significant part of the firmware. Sure, it will need some time to mature, but I’m happy where it’s going. You can find the repository here. I’m using the Lolin D32 PRO board. It has a SD card socket which is super useful for data like AP’s SSID, AP’s password, end server address and port.

I’m also pretty close to 3D printing the enclosure for the first prototype. I still need to get my hands on some basic parts, like tact switches. Those past few days pushed this project quite far.

2020-02-09

Using a buttons inspired by the retro tape recorder means that I have to redesign how I want this device to look like. Few uninspired sketches later…

2020-01-17

I was thinking about another project that would use an old school tape recorder. Then it hit me that the buttons those use are exactly what a project like this needs.

I’m young enough to not have a lot experience with those, but also old enough to remember how did the buttons on those beautiful devices worked. It’s obvious that you can’t rewind a tape and play it back at the same time. You also can’t have it paused and playing at the same time. That means that when you press one button, the one that’s already pressed in pops out. Those keyboards have a great mechanical feel. I might try to cram this project into a cassette tape recorder. There is plenty of space for the electronics. Unfortunately there are some buttons combinations that don’t conform to this simple idea of “only one button pressed at a time”. I think in order to record you press both Play and Record buttons.

I can obviously replicate this behavior with the tact switches. If the user started tracking time on one project by pressing one button, she can just switch to a different project by pressing another button. That would stop tracking time for the previous project and start doing it for the new one.

The idea of piano like key seems nice. I wonder if I could 3D print a keyboard made out of buttons like the one below:

Part of the fun is designing how this device looks, but a cassette tape recorder is a great inspiration. Also, imagine that when you start working on a project, a tape starts spinning (assuming it’s silent, and if I remember correctly there is a hum coming from the recorder when the tape spins). Wouldn’t that look cool? Unnecessary and excessive? Probably.

2019-06-07

Previously I’ve used CircuitLab to draw the schematics. It is pretty good but saving circuits is impossible with a free account. A great alternative is this circuit simulator. As you can see below, the circuit isn’t as aesthetically pleasing.

However, the possibility of saving the circuits to a plain text or in the URL itself is awesome. You can basically copy the text below and import it into the simulator or click this link.

$ 1 0.0000049999999999999996 18.278915558614752 71 5 43
g 176 304 176 352 0
r -32 224 96 224 0 47000
r 176 224 304 224 0 16000
g 304 304 304 352 0
r 304 224 432 224 0 30000
g 432 304 432 352 0
r 432 224 560 224 0 91000
g 560 304 560 352 0
g -48 304 -48 352 0
v -48 304 -48 224 0 0 40 3.3 0 0 0.5
w 112 224 176 224 0
p 96 224 96 304 1 0
g 96 304 96 352 0
w -32 224 -48 224 0
w 96 224 112 224 0
s 176 224 176 304 0 1 false
s 304 224 304 304 0 1 false
s 432 224 432 304 0 1 false
s 560 224 560 304 0 1 false
o 11 64 0 4098 5 0.1 0 1

The live drawing of the signal is great. You can easily see the consistently spread voltage values, corresponding to the specific buttons.

2019-05-30

Lets update the visual concept with the resistors ladder and the ADC pin.

Looks way easier to assemble and expand.

2019-05-30

I’ve planned on explaining how to compute the resistors value. I’ve however found an article that provides all of the necessary information. I need to keep myself in check and focus on what’s most important to finish this project. Even though I believe there is a great value in explaining things from a different perspectives, I decided not to explain the maths behind this in my own words.

The aforementioned article can be found here. I invite you to go through the blog. There is a lot of interesting information there.

The most helpful part for this project is this document. It basically lists all of the resistor values depending on the buttons count. You need to edit document the get the resistors values and in order to do that you have to duplicate it (File -> Create copy).

Setting the No. Switches to 4 and voltage to 3.3V gives the following results. The resistor values are being rounded to a value that’s actually available on the market (nice!). The ADC values are computed too.

Those values look great. They are almost perfectly linear. Pressing the first button grounds the ADC pin - value is 0. Pressing the second button should output ~260, third ~506 and the fourth ~762.

Of course this gets tricker as you add more buttons, since you have to have finer and finer control over the resistances. This won’t be a problem for a project like that - the first version will use only 4 buttons and I can’t imagine the final version using much more.

2019-05-21

With the current configuration, scaling the buttons count is horribly painful. Each button needs its own digital pin. Often used solution is to use a resistor ladder and an ADC.

In the image above the SW2 switch is closed. That makes the current flow through R1 and R2 resistors - that’s basically a voltage divider of 1:1 ratio. That would make a 10-bit ADC read a value of 1024/2 = 512 (well close to…).

Now lets close the SW5 switch instead. That would make a voltage divider with ratio of 1:4. For a voltage value of 3.3V the voltage drop on R1 would be equal to 660mV, while on R2 + R3 + R4 + R5 it would be 2640mV. ADC would output something around 819.

In this solution only one button can be pressed, which is fine for what this keyboard needs to do. What isn’t convenient is that using the same resistors values doesn’t give you a linear numerical values from the ADC. Pushing SW1 would output 1024 (for a 10-bit ADC), pushing SW2 512, SW3 675. Going up from that would mean cramming more into a smaller and smaller values spectrum.

The solution is to compute resistors values to actually compensate for that. Since I want to keep those entries short and simple I’ll explain this next time.

2019-04-14

I’ve explored some hardware options for the buttons. The most cost efficient and comfortable option are the big microswitches. I’ve started designing a frame for buttons like that. The design has to be modular. That means that it has to trivial to expand this frame for any number of buttons.

I’m using Blender to experiment with the design. I achieve the modularity with an Array modifier. Increasing the array’s count modifier just adds the next segment. By default the Array modifier makes it so the elements are perfectly spaced - touching it’s ends.

The frame also has to hide the electronic components… or does it. Maybe those can be used to make the whole project look more interesting? Might explore this idea.

For now the wiring and the resistors are located on the bottom. One thing I’ve failed to consider in the sketch above is connecting the buttons back to the GND line. With this layout those connections would collide with the Vcc. It can be solved by moving the Vcc line more towards the middle of the frame. That way the line could go directly below the switches. That way one of the pins, of the pair close to the GND line, can be easily connected to GND. I will show this change in the next update, assuming I won’t find a better way to do it.

Pressing a specific switch will start time logging for a specific project. That needs to be clear for the user. For now I’m thinking of using LEDs for that. That means that the next design should take that into consideration.