[Noisebridge-discuss] PG&E Tools

Ben Kochie superq at gmail.com
Sun Mar 4 17:09:11 PST 2012


The login process is simple, but there seems to be some redirection
involved with the actual site that has the data I want.

The basic login URL is 'https://www.pge.com/eum/login'  But whenver I
try and post to it, I just get 302'd to the login page.  Maybe I need
to fake the UA.

On Sun, Mar 4, 2012 at 12:21, Daniel Pitts <Daniel at coloraura.com> wrote:
> Taking a csv and stuffing it into a data store is an easy problem to solve, which doesn't require a web scrapper at all. If you need help automating the fetching of the csv, chances are you don't need to scrape, just simulate the login process. curl is a useful unix tool for such a thing.
>
> It is unfortunate that pg&e doesn't have a public restful API for access though. Perhaps a request to them is in order. I've been thinking about that for my own personal purposes.
>
> Thanks,
> Daniel.
>
> Sent from my iPhone
>
> On Mar 4, 2012, at 10:02 AM, Ben Kochie <superq at gmail.com> wrote:
>
>> Yes, that's what I'm talking about.  We need to grab the files there
>> and import them into a datastore of some kind.
>>
>> On Sun, Mar 4, 2012 at 09:33, miloh <froggytoad at gmail.com> wrote:
>>> On Sat, Mar 3, 2012 at 11:43 AM, Ben Kochie <superq at gmail.com> wrote:
>>>> I've spent some time digging around for open source tools for
>>>> accessing PG&E data.  It looks like there is no API or anything useful
>>>> for getting access to the data.  We'll have to write a web scraper to
>>>> pull the data.  Since this isn't something I'm good at, I'm looking
>>>> for some help.  From what I can tell it shouldn't be terribly hard to
>>>> grab the data we need and then feed it into a database.
>>>> _______________________________________________
>>>> Noisebridge-discuss mailing list
>>>> Noisebridge-discuss at lists.noisebridge.net
>>>> https://www.noisebridge.net/mailman/listinfo/noisebridge-discuss
>>>
>>>
>>>
>>> a web scraper?
>>>
>>> for my consumer account I download the usage data in a .csv
>>>
>>> you don't see it for the noisebridge account?
>>>
>>> something like the following:
>>> https://pge.opower.com/ei/app/modules/customer/XXXXXX/bill_periods/export-dialog
>> _______________________________________________
>> Noisebridge-discuss mailing list
>> Noisebridge-discuss at lists.noisebridge.net
>> https://www.noisebridge.net/mailman/listinfo/noisebridge-discuss


More information about the Noisebridge-discuss mailing list