Why use MIDI files?

MIDI files are extremely useful for practicing various musical instruments such as a keyboard or digital piano. You can load the MIDI file into a device such as an tablet or phone and synchronise your instrument with the MIDI file. This allows you to practice effectively using apps such as Synthesia.

Where to find high quality MIDI files?

The issue is that it can be hard to find high quality MIDI files to use for your favourite songs. They are simply not available due to licensing or are sold at absurd prices.

Creating the MIDI file yourself manually is also an extremely time-consuming process.

There are numerous audio-to-MIDI tools available, but they do a poor job at converting a raw audio file to a MIDI file. Often, the output is unusable and requires a significant amount of manual correction, making the effort not worth the time.

Luckily, there are many online piano videos of all of your favourite songs in Synthesia style. In this style, a keyboard is shown from a top down view and the notes flow down the screen as the performer plays.

Here is an example of a Synthesia style piano video on YouTube:

Evil Morty’s Theme – Rick and Morty – For The Damaged Coda by PHianonize

 

We can use a third-party tool called video2midi, created by developer Valentin Sudakov, to process such videos using image recognition and create a near-perfect MIDI file from it. Then we will do make some minor adjustments to create the final perfect MIDI file.

Creating the perfect MIDI file

Below are instructions for creating the perfect MIDI file.

 

(1) Install video2midi

Install video2midi using the instructions provided in the Github repository. Get familiar with the tool and how it works!

(2) Download your video

Download your Synthesia style YouTube video using the video downloader tool of your choice (I use yt-dlp).
Keep in mind that some videos will work better than others based on how much extra noise exists in the video (particle effects, overlays, etc).

(3) Find the correct tempo or BPM for the video

It is important to find the correct tempo for the video. One way of doing this is by finding a link to the music sheet in the description of the video. However, the tempo on the music sheet is often unreliable and will not match the actual tempo of the demonstration in the YouTube video.

Instead, we will use a website called BPM Finder to detect the actual BPM in the video.

Using our example, it seems like the BPM is 72 for this video. Note that the true BPM might be slightly different, but it should be very close to this number!

As it turns out, parts of this song were detected to have 72 bpm and others 66 bpm. In the end, the bpm that will make our measures fit perfectly will turn out to be 69 bpm (the average of these two numbers).

 

(4) Open the video using video2midi.

I will not cover using video2midi in too much detail (as this information can be found on the Github repository page).
There are a lot of settings to tweak here, so read the official documentation and watch example videos linked in the repository.

My notes on tweaking the settings:

• Turn auto close off so you can regenerate the MIDI multiple times as you tweak settings without the application closing itself.
• The start frame and end frame determine the start and end of the playing portion of the video. Give these a small surrounding frame buffer if possible.
• The base octave isn’t too important, we will use a tool to shift all notes down or up a number of octaves if we get this wrong.
• The color map determines how colours are mapped to channels or tracks. Usually, you will have 2 tracks only, so delete (press X) on all but the first two channels. Then click on the first colour square for Channel 1, use the Control+Click hotkey to click on the screen and select the correct colour. Repeat this for the second colour square for Channel 1 (this is used to represent half steps or the black keys). Repeat this for Channel 2 also. I like to have Channel 1 be the left-hand notes and Channel 2 be the right-hand notes.
• The sensitivity controls how sensitive the colour detection is. Use as high of a value as possible while still detecting all the colours correctly. You test this by moving the frame to a frame that has multiple coloured notes being played at the same time and tweaking the sensitivity to see if the detected colour (in the square brackets) changes.
• The roll check can be useful if the black notes are being incorrectly detected as being pressed due to wide rectangular notes for some videos.
• The output tempo is very important. The generated MIDI will always be the same duration as the playing portion of the video, but if your tempo is incorrect, you will find your MIDI will not have the correct beat alignment in each measure/bar, and the total number of measures will also be incorrect. Use the BPM from the previous step to export the MIDI with the correct tempo. You may have to experiment a little to find the ideal BPM to export with based on the alignment of measures.

(5) Import the MIDI file into a MIDI editor

We can use a free online tool to help us do this. Open up the Signal app and click on File -> Open to import your MIDI file.
Right click at the top near the measure indicator and click Add Time Signature. Add the correct time signature from the music sheet or from the beats in each measure in the video.

Next, click on the tracks that contain no notes on the side panel and delete them (in this example, we keep only track 1 and 2 as they are the only tracks that have notes). Rename the first track to Left and the second track to Right to indicate which hand should play those notes.

At this point, you should click on one of the tracks, click the Pencil tool [1], select all notes using Control+A and drag them to the start of measure 1. Also, click Shift+Up or Shift+Down to move all notes to the correct octave if we got our base octave setting wrong from earlier. Repeat this step for all tracks.

You might also notice the first or last measures have incorrect or missing notes if the video used a fade-in or fade-out for the video. If this is the case, you will have to manually correct these measures.

At this stage, you should be able to play your MIDI, and it should sound correct (just like the original video). You can try playing your MIDI and the original video at the same time to ensure they sound exactly the same.

(6) Check the measures!

Our example video has measure markings on the left-hand side. We can see the first note starts at measure 1 and the last notes are played in measure 44 (no notes are played in measure 45). This means there are 43 measures of notes in this song starting at measure 1. The first measure (measure 0) is intentionally left silent, but this is optional.

We want to ensure that our exported MIDI has the same number of total measures. In the Signal app, scroll to the end and confirm the number of measures is equal to the expected value. Also, confirm that various measures have their notes roughly aligned to each beat. They should be almost correct but will not be perfect. The notes should not be overlapping measure boundaries by a significant amount. I recommend checking a measure in the middle of the song side by side with the original video to ensure correctness.

If you have too many or too few total measures or the beats in each measure are misaligned:

• Your time signature may be wrong.
• Your tempo (BPM) when creating the MIDI in the video2midi tool may be wrong.

Try tweaking the time signature in the Signal app, or otherwise go back to the video2midi tool and re-export with a different temp (BPM). Do this until the measures are correct.

(7) Minor adjustment of each note in each measure

As a final step in the Signal app, go through each measure using the Pencil tool [1] and adjust the start and end of each note so it snaps to the grid. You can adjust the grid using the number in the top right corner to allow you to snap to smaller grid lines. You should only need to make very minor adjustments to the note (moving the start or end very slightly). This is because there can be small discrepancies caused by a human playing the song or the image processor’s note detection. Don’t forget to use the original video as a guide!

If you find extreme differences:

• Time signature or tempo (BPM) is wrong.
• Some parts of the song may have a beat that does not align with measures (check the original video!).
• The person playing the song made a mistake (played a note too early / late).
• It is possible the original video had its audio superimposed (in which case there is not much you can do to fix this).

This process essentially dehumanizes the MIDI and makes it a “perfect” MIDI.

Once you have adjusted all the notes, your MIDI file is now complete. You can download it by going to File -> Download MIDI File and begin to use it.

(8) Optional: Export with composer’s intended BPM

In the Signal app, you can also change the BPM to the one that is on the music sheet or the one that is intended by the composer. You can then export the MIDI with this new BPM.

This will change the duration of the MIDI and will cause it to be misaligned with the original video but could potentially make it more authentic (sound as it was intended to be by the composer). The more drastic the difference between the BPM on the music sheet and the BPM you exported the MIDI with, the more drastic the difference in sound and duration.

(9) Optional: Share your MIDI file

After going through all that work, it would be nice to share your MIDI file so the next person doesn’t have to do it all over again!

Unfortunately, I have not found a good free online place to share MIDI files like this with the music community. What I have been doing is uploading the videos I process to this Online Sequencer playlist: https://onlinesequencer.net/playlist/32184

Each sequence description also has a link to a download that contains the MIDI file with channel support (so you can import it into the Synthesia app and get automatic support for left/right hand). This is needed because Online Sequencer does not support multiple channels.

I hope this at least helps some people find the MIDI via Google searches.

Good luck!