Restoring Facebook's Birthday Calendar Export Feature (fb2cal)

Programming511215

Context

Around 20 June 2019, Facebook removed their Facebook Birthday ICS export option.
This change was unannounced and no reason was ever released.

As a heavy user of this feature I was very upset. I use the birthday export feature to be reminded of upcoming birthdays so I can congratulations friends and family. After it became clear that is was not a mistake I decided to write my own scraping tool to restore this functionality for personal use.

This posts includes some of my findings on how I did this.

Where can I find the tool?

If you are simply after the tool, I've open sourced it on Github:
https://github.com/mobeigi/fb2cal

Initial Research

I was sure a scraping solution would work as Facebook still displayed all your friends Birthdays at the /events/birthdays page which is located here:
https://facebook.com/events/birthdays/

Facebook Birthday Events Page

The friend 'bubbles' are grouped by the birthday month. Upon hovering over a user a tooltip with the format Friend Name (DD/MM) is shown which reveals the friends birthday day and month which is all the data we need for our calendar.

Facebook Birthday Hover Tooltip

Scrolling down on the page would dynamically load in more birthday month groups which means AJAX endpoints were being called. Using Chrome Developer Tools I can easily monitor outgoing XHR network requests as I scroll down and trigger the AJAX calls I'd like to replicate.

Chrome Dev Tools XHR Monitor

Querying the Birthday Monthly Card AJAX Endpoint

The Birthday Monthly Card AJAX endpoint we found is responsible for returning the HTML that powers the monthly grouped bubbles pictured above.
We end up with this GET AJAX query (some query parameters have been redacted/snipped):

https://www.facebook.com/async/birthdays/
?date=1567321200
&fb_dtsg_ag=<REDACTED>
&__user=<REDACTED>
&__a=1
&__dyn=<SNIPPED>
&__req=k
&__be=1
&__pc=PHASED%3Aufi_home_page_pkg
&dpr=1.5
&__rev=1000918594
&__s=%3Akqnp2b%3Ay69zbo
&jazoest=28020
&__spin_r=1000918594
&__spin_b=trunk
&__spin_t=1562665727

After some trial and error we notice that the endpoint still returns a valid response as long as we include the following three query parameters: date, fb_dtsg_ag and __a.
Required parameter explanations:

  • The date parameter is an epoch timestamp. The month that the epoch lands in is the month that will be used for the response. So we can pass in any epoch within a particular month to get a response for that month.
  • The fb_dtsg_ag parameter is an async token (CSRF protection token). This token seems to have a lifetime of 24 hours and can be reused between subsequent AJAX requests. It is stored in the source code of the same /events/birthdays page so we can scrape it from there and pass it alongside our AJAX requests.
    "async_get_token":"AQxSD4ZC6HFv74axgbaCIcvRTKp29fSPxI3puZLEFiGfAQ:AQx1pSd8-cFSM6eRRV-VOQ4z_Bc9Hjp_dMYAuRIbhz9sgg"},3515]
  • The __a parameter seems to be a generic action parameter and must be set to 1.

The response from the endpoint looks a little like this:

for (;;);{"__ar":1,"payload":null,"domops":[["replace","#birthdays_pager",false,{"__html":"\u003Cdiv class=\"_4-u2 _tzh _67d4 _4-u8\">\u003Cdiv class=\"_4-u3 _5dwa _5dw9\" id=\"birthdays_monthly_card_1577865600\">\u003Cspan class=\"_38my\">January\u003Cspan class=\"_c1c\">\u003C\/span>\u003C\/span>\u003Cspan class=\"_5dw8\">\u003Cdiv class=\"_tzj\">\u003Ca title=\"John Smith\" href=\"https:\/\/www.facebook.com\/profile.php?id=100000000000000\">John Smith\u003C\/a>, \u003Ca title=\"Other Person\" href=\"https:\/\/www.facebook.com\/otherperson\">Other Person\u003C\/a> and 22 others\u003C\/div>\u003C\/span>\u003Cdiv class=\"_3s3-\">\u003C\/div>\u003C\/div>\u003Cdiv class=\"_4-u3\">\u003Cdiv class=\"_43qm _tzu _43q9\">\u003Cul class=\"uiList _4cg3 _509- _4ki\">\u003Cli class=\"_43q7\">\u003Ca href=\"https:\/\/www.facebook.com\/profile.php?id=100000000000000\" class=\"link\" data-jsid=\"anchor\" data-hover=\"tooltip\" data-tooltip-content=\"John Smith (05\/10)\">\u003Cimg class=\"_s0 _ry img\" src=\"https:\/\/scontent-syd2-1.xx.fbcdn.net\/v\/t1.0-1\/p86x86\/65373832_2374007642378442_5562255821124927488_n.jpg?_nc_cat=100&amp;_nc_oc=AQn2Zu1irnt-mf4JnpRFHNYNxM73Nbw_Mq4xfMRlL1APLPCGUNZ_BXYNtFxEBSNGI18&amp;_nc_ht=scontent-syd2-1.xx&amp;oh=21f0f6ec722b6f36795515d024e32c50&amp;oe=5DE06127\" alt=\"John Smith\" data-jsid=\"img\" \/>\u003C\/a>\u003C\/li>
...

Facebook likes to prefix all AJAX responses with for(;;); as a security measure to prevent JSON hijacking. We can strip this away from the response. The rest of the response is a valid JSON object which we can parse.
We have a lot of useful information in this payload including:

  • Friends Full Name (in the alt and data-tooltip-content fields)
  • Friends Birthday month/day (in the data-tooltip-content field)
  • Link to friends Facebook profile page revealing their vanity_name or profile_id (in the href field)
  • Link to Facebook profile display picture (in the img src field)

We will need all of this information to generate our calendar .ics file except for the Facebook profile display picture.

Parsing the data-tooltip-content

The data-tooltip-content is in the following format (for myself using Facebook locale en_UK):

Firstname Lastname (DD/MM)

There are various problems here! The format is based on the current Facebook users locale.
In other words, the format will change based on the Facebook users selected language.
The date format as well as the ordering of Name/Date can change.

The solution here was to query another AJAX endpoint (https://www.facebook.com/ajax/settings/language/account.php) to retrieve the users locale. Each locale was then mapped to a date format. Finally, we strip away the users name, brackets, right-to-left mark, left-to-right mark and various other unicode characters leaving only the birthday day and birthday month with some separator character in between. It then becomes easy to parse the date using the locale to date format mapping.

Another issue was that Facebook would replace the date with a day name if the Birthday for the friend occurred in the next 7 days relative to the current date (and not the passed in epoch timestamp). So for example if today is the 01/01 and a friends birthday was the next day and that day was a Tuesday, the tooltip content would show John Smith (Tuesday). This logic was easy enough to add once the issue was discovered.

Getting the Facebook entity id

Our calendar .ics file will need a UID (unique identifier) for each friends Birthday event. Otherwise, every time the .ics file is imported into a calendar, duplicate events will be created which is not what we want as we would like to automate the updating process. The obvious candidate for a unique identifier is a Facebook users entity id which is unique per Facebook user, page etc. Vanity names are also unique on Facebook but we decided to not use them as they can change unlike entity ids. Unfortunately, our payload from the Birthday Monthly Card AJAX endpoint does not contain the entity id for every friend. Instead we get a URL to their profile page.

If a friend does not have a vanity name (custom profile page url) setup then we can simply extract the entity id from the id field:

https://www.facebook.com/profile.php?id=1000000000

Otherwise, the problem becomes much more difficult as I did not easily find an endpoint which takes in a vanity name and returns a unique identifier.
The best solution that was discovered was querying the COMPOSER QUERY endpoint (https://www.facebook.com/ajax/mercury/composer_query.php and passing in the following query parameters: value, fb_dtsg_ag and __a. The value parameters is your search query which in our case is the vanity name.

This endpoint is naturally used on Facebook when you are searching for a person to message.

Facebook Composer Endpoint Example Query

This is a typical response for the search term Vanity:

for (;;);{"__ar":1,"payload":{"entries":[{"uid":343154636579910,"photo":"https:\/\/scontent-syd2-1.xx.fbcdn.net\/v\/t1.0-1\/p80x80\/66508358_346692919559415_2878694528699596800_n.jpg?_nc_cat=102&_nc_oc=AQk5rUAaNvGCncHPbREukjvhIPh-4czqqX2YW2DFROQadIHSnpJ60Ce64Xbv6_C3JAs&_nc_ht=scontent-syd2-1.xx&oh=d89d272a15a1883b8e63dd5c54b6e666&oe=5DA4A8D1","type":"page","vertical_type":"PAGE","is_verified":false,"path":"https:\/\/www.facebook.com\/vanity.egy1\/","render_type":"commerce_page","text":"Vanity","is_messenger_user":false,"can_add_to_group_chat":false},{"uid":763331007416193,"photo":"https:\/\/scontent-syd2-1.xx.fbcdn.net\/v\/t1.0-1\/p80x80\/66640625_763331660749461_6540719881631825920_n.jpg?_nc_cat=103&_nc_oc=AQnDWijquwCpVtqrEoTXX09myOfpzMgOoIer5AkIvA0ChbnqhRYyLF_FbbPtu-KDvrQ&_nc_ht=scontent-syd2-1.xx&oh=ca7b8c7486298cca719b5c7723ba671a&oe=5DA9B1AF","type":"page","vertical_type":"PAGE","is_verified":false,"path":"https:\/\/www.facebook.com\/Vanity-763331007416193\/","render_type":"commerce_page","text":"Vanity.","is_messenger_user":false,"can_add_to_group_chat":false},{"uid":381822599086219,"photo":"https:\/\/scontent-syd2-1.xx.fbcdn.net\/v\/t1.0-1\/p80x80\/59295002_381823465752799_7762195369294823424_n.jpg?_nc_cat=104&_nc_oc=AQnyQb5F5Z2uM-Wm_24aM8n_kkRCuzwgQTadPDaKz-l4XBpNmf9llpv1cHucJws_Rig&_nc_ht=scontent-syd2-1.xx&oh=438b063d28bcbc709daf321b1cf0700b&oe=5DE2F6C3","type":"page","vertical_type":"PAGE","is_verified":false,"path":"https:\/\/www.facebook.com\/vanity06\/","render_type":"commerce_page","text":"Vanity","is_messenger_user":false,"can_add_to_group_chat":false},{"uid":1708969295999566,"photo":"https:\/\/scontent-syd2-1.xx.fbcdn.net\/v\/t1.0-1\/p80x80\/62107589_2392281947668294_1669853816914182144_n.jpg?_nc_cat=104&_nc_oc=AQkVkjy5lLZMYHhoSx7LTew1TKljUAjz8wQ5wvkQhocNa0qPhtfXK_A-MPLNjgurSpg&_nc_ht=scontent-syd2-1.xx&oh=85e363302d466723cdcc8b92a45e916e&oe=5DA65F5B","type":"page","vertical_type":"PAGE","is_verified":false,"path":"https:\/\/www.facebook.com\/vanityshopperz\/","render_type":"commerce_page","text":"Vanity Shoppe","is_messenger_user":false,"can_add_to_group_chat":false},{"uid":"100032207542269","type":"user","is_verified":false,"path":"https:\/\/www.facebook.com\/vanity.vanity.963434","names":["Vanity"],"text":"Vanity","subtext":null,"firstname":"Vanity","lastname":"Vanity","photo":"https:\/\/scontent-syd2-1.xx.fbcdn.net\/v\/t1.0-1\/p80x80\/67153939_150032546080329_689664390735069184_n.jpg?_nc_cat=104&_nc_oc=AQmRZXf9yQCqQT3ExGABy0sTKXFye_m5tRHJP9tMqpsrilKMJRMLzWTFKyy4gVccpJU&_nc_ht=scontent-syd2-1.xx&oh=2c1cce028508bdcab3813c0f465d138d&oe=5DA2E6EF","alias":"vanity.vanity.963434","needs_update":true,"non_title_tokens":"vanity","term_to_subtitle":{"vanityvanity963434":"\u0040vanity.vanity.963434"},"index_rank":2,"vertical_type":"USER","prefix_match":"","prefix_length":0,"l_type":"L1_ONLY","match_type":"TAID_NAME","account_status":0,"category":null,"score":345.76179979487,"render_type":"non_friend"},{"uid":"100033679681688","type":"user","is_verified":false,"path":"\/profile.php?id=100033679681688","names":["VA NI TY"],"text":"VA NI TY","subtext":null,"firstname":"VA","lastname":"TY","photo":"https:\/\/scontent-syd2-1.xx.fbcdn.net\/v\/t1.0-1\/p80x80\/67595626_162084538257535_6041839922459967488_n.jpg?_nc_cat=109&_nc_oc=AQmoWz3BpRAep3GE8Qx9qpkNt1VrUaVmuut4uJaLYblxDaFegEvFYXrQYrbkj_AzlqA&_nc_ht=scontent-syd2-1.xx&oh=7f0e168ec9ab6e6fc862d2994726fdb2&oe=5DA4AFF2","alias":"abg.vanity","needs_update":true,"non_title_tokens":"vanity vanity","term_to_subtitle":{"abgvanity":"\u0040abg.vanity"},"index_rank":1,"vertical_type":"USER","prefix_match":"","prefix_length":0,"l_type":"L1_ONLY","match_type":"TAID_NAME","account_status":81,"category":null,"score":278.6572277425,"render_type":"non_friend"}],"end_of_threads":true},"bootloadable":{},"ixData":{},"bxData":{},"gkxData":{},"qexData":{},"lid":"6719806331021820850"}

Note that the search results can return multiple matching entity results including users, pages, apps etc. However, we are guaranteed that our Facebook friend/user with the matching vanity name will appear in the list somewhere. All we have to do here is compare our vanity_name from before with the alias field in the composer query response payload. The matching entry is thus our Facebook friend and we can take the corresponding UID directly off the JSON object.

A consequence of this approach is that we now must perform 1 lookup per Facebook friend to get their entity id. This slows down the script significantly. However, no better solution was found. Third party websites such as findmyfbid.com scrape the users profile page directly to retrieve the entity id from the source code but this approach was profiled as being slower (including via mobile version of Facebook).

Rate limiting note: If the composer query endpoint is hit enough in a short period of times, it seems to somehow restrict the number of entries returned. It seems to limit the query results to only returning results matching friends names, page names exactly. This limitation then disappears over time. So one should be careful how often or quickly they hit this endpoint!

Generating our Calendar ICS File

At this stage, we simply query the Birthday Monthly Card AJAX endpoint passing in epoch timestamps belonging to the first day of every month for a full year (12 months total) and storing the results. We should have all the required fields needed for each friend (Birthday day, Birthday month, Name and UID) so we can generate our Calendar ICS file. This file can then be automatically pushed to the cloud or stored on the local file system for importing into third party applications like Google Calendar.

Final Remarks

This was a fun little project to undertake and a minimum viable product was produced in only a few hours which was nice. It is important to note that these methods do bypass the Facebook API entirely and as a result are against the Facebook TOS. However, as somebody who really wants to tell their friends HAPPY BIRTHDAY! and desperately needs reminders, it was a sin that had to be committed.

Happy Birthday with Cupcakes

Leave a comment

(required)(will not be published)(required)

Comments

Showing 15 comments from 11 commenters.

  • Display picture for Pedro
    Pedro

    hello, I want to have this feature again but I don't understand nothing from programming, it's posible to do this another way? Thank you so much!

    Reply
    • Display picture for Mo Beigi
      Mo Beigi

      Keep an eye on this: https://github.com/mobeigi/fb2cal/issues/15

      I've also been meaning to make a Youtube video with detailed instructions.

      Reply
  • Display picture for Lohith
    Lohith

    I followed the instructions and got this error

    File "fb2cal.py", line 59

    return f'{self.name} ({self.day}/{self.month})'

    ^

    SyntaxError: invalid syntax

    Im not a python developer so I dont understand the problem

    Reply
    • Display picture for Mo Beigi
      Mo Beigi

      Hi! If you're not a dev at all it might be best to wait for this feature: https://github.com/mobeigi/fb2cal/issues/15

      Otherwise, post your issue on Github: https://github.com/mobeigi/fb2cal/issues

      Reply
    • Display picture for Sean Ackley
      Sean Ackley

      Use "python3" or make sure Python 3.7 is installed. Mac comes with python 2.x by default, but using the "pip3" and

      "python3" works great.

      Reply
    • Display picture for Elton Nápoles Núñez
      Elton Nápoles Núñez

      +1

      Reply
  • Display picture for Peter
    Peter

    This is really great!!!

    Reply
  • Display picture for Sean Ackley
    Sean Ackley

    This is simply crazy good programming. Thanks so much! I can't wait to see other innovations.

    Reply
  • Display picture for Marco
    Marco

    Works Great! Thank you very much.

    Definitely a sin that had to be committed.

    Reply
  • Display picture for Leo Krikhaar
    Leo Krikhaar

    Putting the birthdays in manually for the people you actually care about will take 1-hour max, and then your calendar is not clogged up with random events for people you barley know.

    Reply
  • Display picture for Hugo
    Hugo

    if you made this as a chrome extension, I would pay for it :)

    Reply
    • Display picture for Mo Beigi
      Mo Beigi

      Its a good idea but I don't think I have the time.

      Reply
    • Display picture for Allen
      Allen

      Hi Hugo, I'm currently working on building a chrome extension for this. Msg me and I can get you on the wait list

      Reply
  • Display picture for Ghastly Raiderr
    Ghastly Raiderr

    do you guys have a video showing how to do it?

    its too difficult to understand.

    can u please email me?

    Reply
    • Display picture for Mo Beigi
      Mo Beigi

      There is a video here: https://www.youtube.com/watch?v=UnsbV8EJ8-Y

      Hopefully, you can get through all the steps.

      Reply