|Re: Morning AWOL Report for 2 Jan 2018 (1460934)|
|Home > SubChat|
Re: Morning AWOL Report for 2 Jan 2018
Posted by Stephen Bauman on Tue Jan 2 17:05:28 2018, in response to Re: Morning AWOL Report for 2 Jan 2018, posted by BusRider on Tue Jan 2 16:25:31 2018.Can you supply the 3 & 4, E & Q trains?
I thought I already did. If you have a specific question, I'll see if I can answer it.
How were you able to break this data down from the GTFS file?
I read the specs (available on the MTA website) and wrote a program to parse the file. It wasn't easy and took 4 iterations to get something that's manageable.
There are several steps.
1. Download the data. Nobody's archiving the real time data, anymore. You have to apply for a developer's key from the MTA. It's automatic, handled over the web and does not cost anything.
1a. There's a ton of data, so I set up a cloud app to download and save it.
There are several steps in making the data useful.
2. The data is serialized to reduce the byte count. There's a free program called protoc that will convert it into readable a readable text file.
3. The text file is in pseudo xml format. I had to write a program that first fixed up the xml and then converted the xml into multiple csv files.
4. I then wrote a program to import the csv files into a relational database.