RSS
 

Archive for the ‘Programming’ Category

Restoring Facebook’s Birthday Calendar Export Feature (fb2cal)

31 Jul 2019

Context

Around 20 June 2019, Facebook removed their Facebook Birthday ics export option.
This change was unannounced and no reason was ever released.

As a heavy user of this feature I was very upset. I use the birthday export feature to be reminded of upcoming birthdays so I can congratulations friends and family. After it became clear that is was not a mistake I decided to write my own scraping tool to restore this functionality for personal use.

This posts includes some of my findings on how I did this.

Where can I find the tool?

If you are simply after the tool, I’ve opensourced it on Github:
https://github.com/mobeigi/fb2cal

Initial Research

I was sure a scraping solution would work as Facebook still displayed all your friends Birthdays at the /events/birthdays page which is located here
https://facebook.com/events/birthdays/ .

Facebook Birthday Events Page

The friend ‘bubbles’ are grouped by the birthday month. Upon hovering over a user a tooltip with the format Friend Name (DD/MM) is shown which reveals the friends birthday day and month which is all the data we need for our calendar.

Facebook Birthday Hover Tooltip

Scrolling down on the page would dynamically load in more birthday month groups which means AJAX endpoints were being called. Using Chrome Developer Tools I can easily monitor outgoing XHR network requests as I scroll down and trigger the AJAX calls I’d like to replicate.

Chrome Dev Tools XHR Monitor

 

Querying the Birthday Monthly Card AJAX Endpoint

The Birthday Monthly Card AJAX endpoint we found is responsible for returning the HTML that powers the monthly grouped bubbles pictured above.
We end up with this GET AJAX query (some query parameters have been redacted/snipped):

 

After some trial and error we notice that the endpoint still returns a valid response as long as we include the following three query parameters: date , fb_dtsg_ag and __a.
Required parameter explanations:

  • The  date parameter is an epoch timestamp. The month that the epoch lands in is the month that will be used for the response. So we can pass in any epoch within a particular month to get a response for that month.
  • The  fb_dtsg_ag parameter is an async token (CSRF protection token). This token seems to have a lifetime of 24 hours and can be reused between subsequent AJAX requests. It is stored in the source code of the same  /events/birthdays page so we can scrape it from there and pass it alongside our AJAX requests.
    "async_get_token":"AQxSD4ZC6HFv74axgbaCIcvRTKp29fSPxI3puZLEFiGfAQ:AQx1pSd8-cFSM6eRRV-VOQ4z_Bc9Hjp_dMYAuRIbhz9sgg"},3515]
  • The __a parameter seems to be a generic action parameter and must be set to 1.

 

The response from the endpoint looks a little like this:

Facebook likes to prefix all AJAX responses with for(;;);  as a security measure to prevent JSON hijacking. We can strip this away from the response. The rest of the response is a valid JSON object which we can parse.
We have a lot of useful information in this payload including:

  • Friends Full Name (in the  alt and  data-tooltip-content  fields)
  • Friends Birthday month/day (in the data-tooltip-content  field)
  • Link to friends Facebook profile page revealing their vanity_name or profile_id (in the href  field)
  • Link to Facebook profile display picture (in the img  src  field)

 

We will need all of this information to generate our calendar .ics file except for the Facebook profile display picture.

 

Parsing the data-tooltip-content

The data-tooltip-content is in the following format (for myself using Facebook locale en_UK ):

There are various problems here! The format is based on the current Facebook users locale.
In other words, the format will change based on the Facebook users selected language.
The date format as well as the ordering of Name/Date can change.

The solution here was to query another AJAX endpoint ( https://www.facebook.com/ajax/settings/language/account.php ) to retrieve the users locale. Each locale was then mapped to a date format. Finally, we strip away the users name, brackets, right-to-left mark, left-to-right mark and various other unicode characters leaving only the birthday day and birthday month with some separator character in between. It then becomes easy to parse the date using the locale to date format mapping.

Another issue was that Facebook would replace the date with a day name if the Birthday for the friend occurred in the next 7 days relative to the current date (and not the passed in epoch timestamp). So for example if today is the 01/01 and a friends birthday was the next day and that day was a Tuesday, the tooltip content would show  John Smith (Tuesday) . This logic was easy enough to add once the issue was discovered.

Getting the Facebook entity id

Our calendar .ics file will need a UID (unique identifier) for each friends Birthday event. Otherwise, every time the .ics file is imported into a calendar, duplicate events will be created which is not what we want as we would like to automate the updating process. The obvious candidate for a unique identifier is a Facebook users entity id which is unique per Facebook user, page etc. Vanity names are also unique on Facebook but we decided to not use them as they can change unlike entity ids. Unfortunately, our payload from the Birthday Monthly Card AJAX endpoint does not contain the entity id for every friend. Instead we get a URL to their profile page.

If a friend does not have a vanity name (custom profile page url) setup then we can simply extract the entity id from the id field:

Otherwise, the problem becomes much more difficult as I did not easily find an endpoint which takes in a vanity name and returns a unique identifier.
The best solution that was discovered was querying the COMPOSER QUERY endpoint ( https://www.facebook.com/ajax/mercury/composer_query.php and passing in the following query parameters: value , fb_dtsg_ag and __a. The  value parameters is your search query which in our case is the vanity name.

This endpoint is naturally used on Facebook when you are searching for a person to message.

Facebook Composer Endpoint Example Query

This is a typical response for the search term Vanity:

Note that the search results can return multiple matching entity results including users, pages, apps etc. However, we are guaranteed that our Facebook friend/user with the matching vanity name will appear in the list somewhere. All we have to do here is compare our vanity_name from before with the alias field in the composer query response payload. The matching entry is thus our Facebook friend and we can take the corresponding UID directly off the JSON object.

A consequence of this approach is that we now must perform 1 lookup per Facebook friend to get their entity id. This slows down the script significantly. However, no better solution was found. Third party websites such as findmyfbid.com scrape the users profile page directly to retrieve the entity id from the source code but this approach was profiled as being slower (including via mobile version of Facebook).

Rate limiting note: If the composer query endpoint is hit enough in a short period of times, it seems to somehow restrict the number of entries returned. It seems to limit the query results to only returning results matching friends names, page names exactly. This limitation then disappears over time. So one should be careful how often or quickly they hit this endpoint!

Generating our Calendar ICS File

At this stage, we simply query the Birthday Monthly Card AJAX endpoint passing in epoch timestamps belonging to the first day of every month for a full year (12 months total) and storing the results. We should have all the required fields needed for each friend (Birthday day, Birthday month, Name and UID) so we can generate our Calendar ICS file. This file can then be automatically pushed to the cloud or stored on the local file system for importing into third party applications like Google Calendar.

Final Remarks

This was a fun little project to undertake and a minimum viable product was produced in only a few hours which was nice. It is important to note that these methods do bypass the Facebook API entirely and as a result are against the Facebook TOS. However, as somebody who really wants to tell their friends HAPPY BIRTHDAY! and desperately needs reminders, it was a sin that had to be committed.

Happy Birthday with Cupcakes

 
4 Comments

Posted in Programming

 

Discord Discriminator Farming

30 Apr 2018

This post outlines how I got the Discord discriminator that I wanted.

This process is old and is no longer required as Discord now allow you to change your discriminator via Discord Nitro.

What are discriminators?

Discord is a free modern voice and text chat application. Discord uses usernames to identify its users. However, instead of a username becoming unavailable after just 1 person uses it, Discord allows 9999 people to share the same username. It does this by using the combination of a name (unicode string) and a discriminator (4 digits) as a Discord tag.  SomeUserName#1234 is an example of a Discord tag. This is a great idea (and more services should use it) as it allows many extra people to use their first name or a common alias as their username.

Each Discord account still has an underlying Discord user ID (i.e. 356005924638224307 ) that can is echo’d if you type in  \@SomeUserName#1234 into any text channel. This is typically hidden from most users and used by developers when making plugins and bots.

Why do discriminators matter?

They don’t but just like usernames, discriminators are visible and thus can be considered to be a cosmetic name tag attached to your account. This is why people want specific Discord discriminators. Because it looks nice! User#0001  or User#1337  look a lot nicer than User#6519. For a long time, Discord has been strictly against scripts/bots which were designed to change a users Discriminator. This is because they believed the aesthetics of discriminators did not matter and their only purpose was to allow more people to share the same usernames. This statement was somewhat unfair given that almost all Discord devs had manually changed their discriminators!

Writing a Discord Discriminator Farmer

Back in mid 2017, I decided I wanted a new fancy Discord discriminator to replace my randomly generated one of 8471.

There is only one way to change your discriminator. Discord offer a name change feature in which you can change your username but not your discriminator. However, what happens if the username you want is already taken by someone with the same discriminator as you? Two users can’t have the same Discord tag. In this case, Discord changes your username (assuming all 9999 discriminator for that username aren’t taken) and then randomly generates a new discriminator for you!

This is the key behaviour that I used to write some python scripts to farm Discord discriminators.

Step 1: Gathering a list of usernames and discriminators

The first step in the process was to write a script ( get_discrim.py) to store a large number of Discord usernames and discriminators belonging to existing users. Recall that we needed to change our username to a new username that already had a user with our current discriminator. For example, if our Discord username was  SomeUserName#6513 and a user existed called  Tony#6513, we could change our username to Tony and because  Tony#6513 already exists, Discord would generate a new random Discriminator for us.

Overall, this script was fairly simple to make. We simply made a new Discord account and joined a lot of guilds (aka servers) with very high member counts. We then uses the Discord API to return a list of all members in the guild that were currently online. By joining a few massive guilds like /r/Overwatch and /r/PUBATTLEGROUNDS, we had access to over 60000+ Discord tags which meant we had an existing username for 99% of discriminators. Our script stored the results as a dict which was dumped as a pickle so our second script could use it. This file was regenerated every 15 minutes to ensure we wouldn’t get stuck (in the rare case where our dump contained no matching usernames for a particular discriminator).

Step 2: Farming them Discriminators

The second script ( farm_web.py) would authenticate with Discord, load in our pickle from our first script and begin changing usernames. However, there was a problem. Discord initially allowed you to change your username as many times as you wanted. Then they restricted username changes to once per hour. Then once per two hours. Then once per day which is what the “secret” time window was when I was testing. This was to combat people doing exactly what I did. Now, given that I wanted a very small subset of target discriminators (22 total) out of a possible 9999, this would not do. As there was no way to get around this username change time limit I was forced to use multiple accounts and change each of their usernames daily.

This worked well initially before I ran into another issue. IP rate limiting. Discord would rate limit my servers IP address causing a lot of the username change API requests to fail. I overcame this quite easily by spanning out my name changes throughout the day rather than making them all at once.

Another issue was the fact that each account needed an Authorization Token to authenticate with the API. I ended up manually fetching and storing the authorization tokens for all the accounts I used by logging into each account, filling in a captcha if one was presented and then retrieving the authorization token from the browsers local storage. As long as you did not log out, the authorization token remained valid indefinitely (this is why you can stay signed into the same Discord account forever on the same machine/browser).

Finally I was able to run my script successfully with about ~150 accounts at once. Meaning I had 150 new discriminators generated over a 24 hour period. Again, this isn’t a huge amount but it was enough to make bruteforcing feasible.

Discord Account List

Once a name changed resulted in a new random discriminator that was in my target list, that thread would end and an entry would be written to my log file to alert me.

Step 3: One final name change

Over 2 weeks later running my scripts 24/7, I finally had 1 alert letting me know that a target discriminator was found. I really liked the result so I decided I would keep it and turn off my scripts. However, the account with the final discriminator did not have the username I wanted.

The final step involved changing the username to the username that I wanted to use. This part is important, you had to ensure the username you were changing your name to DID NOT have a discriminator that matched your new discriminator. Otherwise, Discord would just give you a randomly generated discriminator. This was easy enough, all you had to do was try to add a user via the friend system and see if the friend request was send successfully indicating the account existed or not indicating the account did not exist and it was safe to proceed. For example, if my final account with my target discriminator was  RandomUserName#1337  and I wanted my tag to be  MyName#1337, I would send a friend request to  MyName#1337 to see if that tag existed. If it did not, I could proceed and get it for myself! Otherwise, you would unfortunately be out of luck.

Changes to Date

After I stopped running my scripts, Discord eventually increased the username change time window and enforced harsher IP rate limiting. Finally, they allowed users to change their discriminator via Discord Nitro although most of the good ones are probably taken by now.

Source Code

The source code for this project is available here:
https://github.com/mohammadg/Discord-Farmer

 
2 Comments

Posted in Programming

 

Making my First libGDX Game

15 Apr 2018

Recently I decided to create a simple Android game and it was much easier than I thought it would be!
I had never made an Android game before but I had made a few Android apps as well as a JOGL game which is great as it meant I had a good idea of what was happening behind the scenes.

Game development frameworks like libGDX often make it easy to create games because they handle a lot of the low level work. libGDX is more or less a wrapper over OpenGL and allows you to make raw OpenGL API calls if you so desire. Even though libGDX is very simple to use, I would still recommend making a simple game with raw OpenGL to fully grasp what is happening behind the scenes.

Picking a game to make

The first step in making a game is deciding what you actually want to make. For my first game I thought I’d stick to something simple that other developers had made tutorials on. Turns out that Flappy Bird clones are the Hello World App’s of the game development world. As I own a green cheek conure called Siavash it was only reasonable I made a clone called Flappy Siavash.

Brent Aureli’s Tutorial

I’m not going to go through the actual game making process in this post.
I ended up following this tutorial by Brent Aureli up to around part 7 before I got a hang of things and started to finish off the game myself.

Features

My game ended up having the following features:

  • Main Menu Screen (very basic with a play button)
  • Custom sprite (coloured to match my bird Siavash)
  • Obstacle crash animation (rotating into nose dive into ground)
  • Obstacle crash screen flash
  • Using cages as the obstacle
  • Number of cages active at any given time can be changed
  • Fade in/Fade out for various text/sprites/etc
  • Pause menu to pause game
  • Game over menu with option to play again
  • High score functionality (using local storage)
  • Works and tested on Android, Java, Web
  • Uses custom distance field font with fragment/vertex shader
  • Scrolling backgrounds
  • Randomised obstacles
  • Endless mode

 

Gameplay

Flappy Siavash Gameplay

 

Tips

Here are some tips which may help you when creating your own libGDX game:

  • Make sure you dispose resources (textures etc) when you are done using them or you will have memory leaks.
  • Don’t remake the same Texture etc all the time, store the Texture somewhere (in the class or put in in an asset manager and retrieve it whenever needed)
  • If possible, keep assets with widths/heights that are powers of 2. Assets seem to fail to load correctly with GWT (Google Web Toolkit) otherwise.
  • Organise assets in subfolders (shaders/images/sounds/sprites)! Don’t dump everything into the same folder.
  • When updating your states, always use dt (delta time) to scale your changes appropriately.
  • Place sprites in your word using coords relative to the cameras viewport width/height.
  • Keep a helper class with various static methods useful for performing common tasks (do the same with a debug class for debugging).

 

Source Code

The code from my project is available on Github if you are interested:
https://github.com/mohammadg/Flappy-Siavash

Play Online

Play Online At:
https://mobeigi.com/blog/projects/flappysiavash/

Play Store App

I decided to release this app to the Google Play Store to learn about the Play Store publishing process.

Available at:
https://play.google.com/store/apps/details?id=com.mohammadg.flappysiavash

My Next Project

I am now working on a much more complicated/unique game which I plan to polish up.
This project will likely take much, much longer but I will also learn a lot along the way.

 
No Comments

Posted in Programming

 

Run Adobe Audition in the Background to Reduce your Microphone’s Background Noise

08 Apr 2017

Recently I have been looking for a way to reduce the background noise my microphone produces. I own a Blue Yeti Microphone mounted on a RODE Arm Stand and I like to keep my microphone fairly far away so its not in my face and doesn’t distract me while I record audio or play video games.

However, at this distance, the microphone unfortunately picks up a lot of background noise including computer fans, outside noises and even small things like picking up/putting down a cup of water.

Part of the Solution

To solve this I found a wonderful video by SaaiTV linked below. I have built onto this solution to make it better but the first thing you should do is follow the Youtube video tutorial and come back here to continue. Keep in mind I am using Adobe Audition CS6 and would recommend you use the same version (it will help later on in this tutorial).

The Problem

If you followed the instructions in the video and are happy with the result, you may want to keep the noise reduction effect so its always on. For recording audio and small tasks, you can simply run Adobe Audition and open your saved session and then close it when you are done. If this is all you want to do then this post won’t help you.

However, if you want to keep the noise reduction effect 24/7 so you can take advantage of it all the time, I will tell you how to run Adobe Audition when your computers boots up in the background so its out of the way.

Rest of the Solution

When you run Adobe Audition in the background keep in mind it will always be running. On my machine (which is quite good), it used 150MB of RAM and 1-3% CPU constantly.

Task Manager Adobe Audition CPU/Memory Usage

This is no issue for me at all but might reduce performance significantly on other machines.

Now, the first thing you want to do is download a program called AutoHotKey from https://autohotkey.com/

Once installed, remember the full path to the  AutoHotkey.exe executable. We will need it later.

In my case, the full path is:

Now, pick a folder on your computer to store a new AutoHotKey script file ( .ahk  file).
In this example I’ll pick:

We are going to put a new file here called  adobe_audition_microphone.ahk. Download the file I have prepared and copy and paste it in this folder: Download Link

Now, open this file using Notepad (or your favourite text editor). You should see this text:

There are a few adjustments you need to make.
Firstly, replace ADOBE_AUDITION_PATH with the path to your Adobe Audition executable. Make sure to keep the quotes around the path intact.
Mine was located at:

Next, replace SESX_PROFILE_FILE with the path to your Adobe Audition session file. This is the session file you should have saved when following the video tutorial. You can copy this file to the same folder as the .ahk  file we are modifying right now.

I called my file Microphone_Noise_Reduction.sesx and moved it to:

Finally, you will need to change  #32770 and  audition5  if you are NOT using Adobe Audition CS6. You can find the correct values to use using AutoHotKeys Window Spy tool. I will not cover that in this post but leave a comment below if you have trouble with this.

Your adobe_audition_microphone.ahk  file should now be complete.
This is what mine looks like when completed:

Make sure to save your changes before exiting your text editor.
Now, perform a little test by double clicking your  adobe_audition_microphone.ahk  file.
If everything is working, it should open up Adobe Audition, and then hide the window once its open.
It should not appear in your taskbar as a minimised window.
Make sure you see Adobe Audition running in Task Manager.

Now go to Windows Sound options, go to the Recording tab, right click on your Virtual Audio Cable Line, click properties, go to the Listen tab and press Listen to this device. You should now hear your microphone output as processed by Adobe Audition. If you do not, then you have done something wrong.

Windows Sounds Listen to this Device

 

The last step is to make our AutoHotKey script ( C:\MyAhkFolder\adobe_audition_microphone.ahk ) start with Windows so your microphone’s input is always being processed and output to your virtual audio cable.
There are plenty of tutorials on how to do this on the internet.

Here is an easy one to follow:
https://www.autohotkey.com/docs/FAQ.htm#Startup

Good luck!

 
13 Comments

Posted in Programming

 

Simple PHP File Download Script

18 Dec 2015

So I recently added a download.php script to my website so that I could force downloads of files instead of having users access them through an indexed directory or through their browser.

I found various scripts online but none of them were as clean as I’d have liked them to be so I wrote my own simple script after a bit of research.

In my setup, the download.php file sits at the root of my website and the filevault folder sites one level higher on the web server. This setup ensures users cannot hotlink to files or directly access them, the script must be used. A benefit of this is that you can add restrictions like allowing a file to be accessed by people from a particular country or by those who have a certain cookie set. If you do not have access to the directory above your websites root directory then you are forced into putting your filevault at the websites root directory.

This is the simple PHP File download script:

Usage

The following link would force the download of that_file.txt

 

Demo

You can also download the above script (using the script!):
Download download.php Script

 
8 Comments

Posted in Programming

 

How to fix your League of Legends Registry Paths (OP.GG fix)

30 Nov 2015

Amumu Sad Mummy

Artist: Awskitee

The issue

If you play League of Legends and use any third party tools (to record replays, enhance gameplay, among other things :p) then you have probably run into programs that ask for the path of your League of Legends directory. Fortunately, some programs can auto detect this using the Windows registry. However, if you have moved your League of Legends folder to another folder, reinstalled/upgraded Windows or used a registry cleaner in the past then its possible that your registry entries are corrupt. This means they either don’t exist or don’t point to the correct path.

This is a common issue for most people who cannot watch OP.GG replays as the OP.GG replay batch files rely on the RADs path to function correctly. You might have run into this error:

KR: LOL 경로를 자동으로 찾을 수 없습니다. 도움말에서 관전하기 문제 해결을 보시면 100% 해결 될 수 있습니다. 100% 해결 될 수 있으니, 채팅방에서 괜히 사서 고생해서 물어보지마세요!!!!!!!
EN: Cannot found LOL directory path for automatic. Please see our spectate help page: http://oce.op.gg/help/observer

 

OP.GG has a section on their website that can allow you to set the path manually (which will be reflected in downloaded batch files) but this must be done for each region and every now and again which is not ideal. For the OCE server, the instructions are under the League of legend.exe file can not be found header at http://oce.op.gg/help/observer/

 

A permanent solution

I decided to write a script that can fix this issue for third party applications and web apps.
Simply copy and paste the following code into a text file (.txt) using a text editor of your choice (ie notepad).
Then save the file with a .bat extension (call it anything you like such as fix.bat).
Now simply right click the file and Run as Administrator .

Finally, follow the on screen instructions and choose your system architecture (32 bit/64 bit) and enter the path to your LoL folder. If you do not know your system architecture then run the program twice and use both options.

Download Link:
LoL RADS Registry Fixer.bat

Batch script:

 
101 Comments

Posted in Programming