Get-FBPost cannot see posts older than November 2011

Dec 27, 2011 at 11:52 PM

I connected your code module with the AppId I already had, which means I did this to initialize it:

Show-FBConnectionDialog -AppId mypersonalappid

 Then, I ran the following to see if Get-FBPost would go back to 2009 when I first started using Facebook:

Get-FBPost | Sort-Object created_date | Format-Table created_date,updated_date,message -AutoSize

To my surprise, the "oldest" post retrieved was from November 2011

Why is that?

Should I try any of the "raw" Get-FBzzzzzzz calls?  If so, which one(s)?

 

Coordinator
Dec 28, 2011 at 6:52 PM

How much data did you get back from the command? It is a known issue that FacebookPSModule does not currently support the Facebook Graph API Paging capability, so you will only get the first “page” of data. If this is a critical capability for your organization, I can consider adding it.

FacebookPSModule’s support for the “-AppId” field is currently rudimentary. I already plan to improve this in the next release. Why are you using an alternate AppId? I also note that your message specifies “-AppId mypersonalappid”, but I assume you meant “$mypersonalappid” or the like – the literal “mypersonalappid” is not a real appid.

Note that the correct field names are “updated_time” and “created_time”.

If you could provide more data on your scenarios, this would help me prioritize future work.

Thanks,

Jon Newman

Dec 29, 2011 at 1:37 AM
Edited Dec 29, 2011 at 1:39 AM

Jon: I used my own personal AppId (a pure number as you menioned) because (a) I don't feel comfortable using someone else's AppId which basically gives them full access to my Facebook "stuff" and (b) I already had an AppId from experimenting with the Facebook SDK (also here on Codeplex).

What I'm seeing is possibly a 30-day "sliding window" of posts, despite using various combinations of:

Get-FBFeeds

Get-FBPost

Get-FBPosts

Get-FBObjectData

I also turned on Timeline in Facebook about 3 days agao and have been busy "pruning" old posts (from 2009 onward) and was looking for a quick way to use PowerShell to possibly perform a quicker, mass pruning of all old posts older than some date that I establish through code:

  $dateCompare = New-Object -TypeName 'System.DateTime' -ArgumentList 2009,6,1

Get-FBPost | Where-Object{ $_.created_time -lt $dateCompare} | Format-Table created_time,updated_time,message -AutoSize 

However, until I can retrieve posts older than November 27, 2011, I'm dead in the water.

Coordinator
Dec 29, 2011 at 6:57 AM

Perhaps my understanding is incorrect regarding what it means for a user to grant access to an application. I had believed that the application only has access if it has the access token granted by Connection Dialog, and that the application has to remember the access token from when it was presented to the RedirectUri. Is this wrong? If so, I will need to reconsider some aspects of FacebookPSModule's security architecture.

I have never seen this 30-day-window behavior. Perhaps this is connected with switching to Timeline? It certainly wouldn't be the first time Facebook has changed API behavior. Again, how many objects are you receiving in that window? Also: If you change $VerbosePreference/$DebugPreference to "Continue", do you see a reference to paging objects in the debug/verbose streams?

Your post-pruning scenario is interesting, but I'm not sure that automated deletion of posts will work for posts not created by your application. Per https://developers.facebook.com/docs/reference/api/post/:

Delete

You can delete a post as long as your application created the post. You delete a post by issuing an HTTP DELETE request to the POST_ID object with publish_stream permission.

If the delete is successful, you get the following return.

Description Type
If the delete succeeded boolean
Dec 29, 2011 at 3:08 PM

CodePlex is preventing me from replying to your post (IsError=True keeps popping up) in full, so I'm going to try breaking it up into multiple replies to see if I can do that.

Dec 29, 2011 at 3:09 PM

Reply part 1 (the script):

Clear-Host

Write-Host $VerbosePreference
$VerbosePreference = 'Continue'
Write-Host $VerbosePreference

Write-Host $DebugPreference
$DebugPreference = 'Continue'
Write-Host $DebugPreference

Get-FBPosts | Format-Table type,created_time,PostID -AutoSize
Dec 29, 2011 at 3:12 PM

Still cannot post the huge script output as a reply.  I get the following error from CodePlex when I try.  I'll try breaking it up into 10 or more smaller pieces, but I warn you it will be "death by a 1000 cuts" to do so:

{"IsError":true}
Dec 29, 2011 at 3:15 PM

Reply page 2 of 42:

DEBUG: Convert-FBJSON                 System.String Fred Morrison

DEBUG: Convert-FBJSON               key offset

DEBUG: Convert-FBJSON                 System.Int64 0

DEBUG: Convert-FBJSON               key length

DEBUG: Convert-FBJSON                 System.Int64 13

DEBUG: Convert-FBJSONHash           hash not containing data value

DEBUG: Convert-FBJSONHash       hash not containing data value

DEBUG: Convert-FBJSON        key picture

DEBUG: Convert-FBJSON           System.String https://fbcdn-photos-a.akamaihd.net/photos-ak-snc1/v43/11/145634995501895/app_1_145634995501895_4870.gif

DEBUG: Convert-FBJSON         key link

DEBUG: Convert-FBJSON           System.String http://developers.facebook.com/tools/explorer/

DEBUG: Convert-FBJSON         key caption

DEBUG: Convert-FBJSON           System.String A tool to help you browse objects within the Facebook Graph API, manage permissions, obtain access tokens and generally learn how it all works.

DEBUG: Convert-FBJSON         key actions

DEBUG: Convert-FBJSON           JSONArray

DEBUG: Convert-FBJSON             JSONObject

DEBUG: Convert-FBJSON             key name

DEBUG: Convert-FBJSON               System.String Comment

DEBUG: Convert-FBJSON             key link

DEBUG: Convert-FBJSON               System.String http://www.facebook.com/1400310498/posts/2356090707271

DEBUG: Convert-FBJSONHash         hash not containing data value

DEBUG: Convert-FBJSON             JSONObject

DEBUG: Convert-FBJSON             key name

DEBUG: Convert-FBJSON               System.String Like

DEBUG: Convert-FBJSON             key link

DEBUG: Convert-FBJSON               System.String http://www.facebook.com/1400310498/posts/2356090707271

DEBUG: Convert-FBJSONHash         hash not containing data value

DEBUG: Convert-FBJSON         key type

DEBUG: Convert-FBJSON           System.String link

DEBUG: Convert-FBJSON         key created_time

DEBUG: Convert-FBJSON         time value

DEBUG: Convert-FBJSON         key updated_time

DEBUG: Convert-FBJSON         time value

DEBUG: Convert-FBJSON         key comments

DEBUG: Convert-FBJSON           JSONObject

DEBUG: Convert-FBJSON           key count

DEBUG: Convert-FBJSON             System.Int64 0

DEBUG: Convert-FBJSONHash       hash not containing data value

DEBUG: Convert-FBJSONHash     hash not containing data value

DEBUG: Convert-FBJSON         JSONObject

DEBUG: Convert-FBJSON         key id

DEBUG: Convert-FBJSON           System.String 1400310498_2353423240586

DEBUG: Convert-FBJSON         key from

DEBUG: Convert-FBJSON           JSONObject

DEBUG: Convert-FBJSON           key name

DEBUG: Convert-FBJSON             System.String Fred Morrison

DEBUG: Convert-FBJSON           key id

DEBUG: Convert-FBJSON             System.String 1400310498

DEBUG: Convert-FBJSONHash       hash not containing data value

DEBUG: Convert-FBJSON         key story

Dec 29, 2011 at 3:16 PM

Page 3 of 42:

DEBUG: Convert-FBJSON           System.String Fred Morrison likes a link.

DEBUG: Convert-FBJSON         key story_tags

DEBUG: Convert-FBJSON           JSONObject

DEBUG: Convert-FBJSON           key 0

DEBUG: Convert-FBJSON             JSONArray

DEBUG: Convert-FBJSON               JSONObject

DEBUG: Convert-FBJSON               key id

DEBUG: Convert-FBJSON                 System.Int64 1400310498

DEBUG: Convert-FBJSON               key name

DEBUG: Convert-FBJSON                 System.String Fred Morrison

DEBUG: Convert-FBJSON               key offset

DEBUG: Convert-FBJSON                 System.Int64 0

DEBUG: Convert-FBJSON               key length

DEBUG: Convert-FBJSON                 System.Int64 13

DEBUG: Convert-FBJSONHash           hash not containing data value

DEBUG: Convert-FBJSONHash       hash not containing data value

DEBUG: Convert-FBJSON        key picture

DEBUG: Convert-FBJSON           System.String https://s-external.ak.fbcdn.net/safe_image.php?d=AQCjAQCzKcQRsDab&w=90&h=90&url=http%3A%2F%2Fwashingtonexaminer.com%2Ffiles%2F199b1d52f92d5e1c020f6a70670025da_0.jpg

DEBUG: Convert-FBJSON        key link

DEBUG: Convert-FBJSON           System.String http://washingtonexaminer.com/news/2011/12/fbi-shooting-range-popular-deer-hangout/2035076

DEBUG: Convert-FBJSON         key name

DEBUG: Convert-FBJSON           System.String FBI shooting range is popular deer hangout | Washington Examiner

DEBUG: Convert-FBJSON         key description

DEBUG: Convert-FBJSON           System.String Call it a playground for Bambi and G-Men, where imaginary criminals are hunted and deer are the spectators.               The 547-acre FBI Academy, where some of the nation's best marksmen fire off more than 1 million bullets every month, happens to be one of the safest places for deer during huntin...

DEBUG: Convert-FBJSON         key icon

DEBUG: Convert-FBJSON           System.String https://s-static.ak.facebook.com/rsrc.php/v1/yD/r/aS8ecmYRys0.gif

DEBUG: Convert-FBJSON         key actions

DEBUG: Convert-FBJSON           JSONArray

DEBUG: Convert-FBJSON             JSONObject

DEBUG: Convert-FBJSON             key name

DEBUG: Convert-FBJSON               System.String Comment

DEBUG: Convert-FBJSON             key link

DEBUG: Convert-FBJSON               System.String http://www.facebook.com/1400310498/posts/2353423240586

DEBUG: Convert-FBJSONHash         hash not containing data value

DEBUG: Convert-FBJSON             JSONObject

DEBUG: Convert-FBJSON             key name

DEBUG: Convert-FBJSON               System.String Like

DEBUG: Convert-FBJSON             key link

DEBUG: Convert-FBJSON               System.String http://www.facebook.com/1400310498/posts/2353423240586

DEBUG: Convert-FBJSONHash         hash not containing data value

DEBUG: Convert-FBJSON         key type

DEBUG: Convert-FBJSON           System.String link

DEBUG: Convert-FBJSON         key created_time

DEBUG: Convert-FBJSON         time value

DEBUG: Convert-FBJSON         key updated_time

Coordinator
Dec 29, 2011 at 7:58 PM

Yes, "encountered data/paging pair" is the indicator I was looking for. Could you run "(Get-FBFeed).Count" to determine how many objects were returned?

Coordinator
Dec 29, 2011 at 8:03 PM
I deleted this post because it contains your access token.

From: [email removed]
To: [email removed]
Date: Thu, 29 Dec 2011 07:14:32 -0800
Subject: Re: Get-FBPost cannot see posts older than November 2011 [FacebookPSModule:284292]

From: fmorriso
Reply Page 1 of 42:
SilentlyContinue
Continue
SilentlyContinue
Continue
DEBUG: Get-FBPost: id me
DEBUG: Get-FBAssociation: type Posts id me
DEBUG: Get-FBRawData: query me/Posts
DEBUG: Get-FBConnection: entering
DEBUG: Get-FBConnection: client not cached
VERBOSE: Reading access token from file C:\Users\fpmorrison\AppData\Local\FacebookPowerShellModule_CachedToken.txt
DEBUG: Read-FBConnection: Read access token ...
Dec 29, 2011 at 8:09 PM
jonn_msft wrote:

Yes, "encountered data/paging pair" is the indicator I was looking for. Could you run "(Get-FBFeed).Count" to determine how many objects were returned?

Write-Host (Get-FBFeed).Count

gives a value of 25

Is that the default limit=n value when querying Facebook?  How can I specify a higher limit, say 400?

Coordinator
Dec 30, 2011 at 7:58 AM

I confirm that when you modify Get-FBAssociation to contain something like

        if ($Fields -ne $null)
        {
            $query = $query + "?fields=$($Fields -join "","")"
            $query = $query + ",limit=100" # BUGBUG TEST
        }
        else
        {
            $query = $query + "?limit=100" # BUGBUG TEST
        }

you get more than 25 entries. https://developers.facebook.com/docs/reference/api/user/ says that the "home" and "feed" connections contain just 25 posts, which presumably is a default. I can try to work in a way to modify the default Limit, perhaps as a connection parameter. Real paging support will be more complicated.

Your particular search could benefit from the "until" parameter (https://developers.facebook.com/docs/reference/api/), which I also do not yet support.

Dec 30, 2011 at 3:08 PM

Your suggestion works up to a point.

Also, there is a bug in your suggested script when $Fields is null.  The separator character should be ampersand, not comma.

When I set limit to 100, 200 or 300, I get that man items +2 (I assume these are previous and next paging urls?).

When I set limit to 400, I get back 362 items, which I assume means I get 360 plus the 2 for "paging".

My adjusted Facebook.ps1 script (the part I changed, not the entire function):

$sbr = New-Object -TypeName 'System.Text.StringBuilder' -ArgumentList "$Id/$Type"       
[char] $sepchar = '?'
if ($Fields -ne $null)
 {            
    $sbr.Append("?fields=$($Fields -join "","")")
    $sepchar = '&' # set the correct separator character to ampersand, not comma
}
$sbr.Append($sepchar)
$sbr.Append('limit=400')  # curiously, the maximum usable limit value is about 360
$query = $sbr.ToString()

My test script:

$posts = Get-FBPosts
Write-Host $posts.Count

The output from my test script:

362

 

Dec 30, 2011 at 4:22 PM
Edited Dec 30, 2011 at 4:25 PM

When I add

&since=2010-01-01

and/or

&until=2010-12-31

to the query, I only get one post back, for November 5, 2010, so I suspect the Facebook Graph API has some peculiar "What Mother Never Told You About..." gotchas in it, probably part of some "prevent denial of service attacks" on Facebook that is part of their "throttle" mechanism (or whatever they choose to call it).  If that's what it is, I can't really blame FB for trying to protect itself, but the bottom line is, if I want to run a query such as the following, it won't give me back the complete set of results I'm looking for:

me/Posts?fields=type,created_time&limit=3000&since=2010-01-01&until=2010-12-31

All I get is this one entry, when I know for certain from using FB interactively that there are dozens of them (I can see them when I use Facebook interactively):

status 11/5/2010 8:44:13 AM 

Coordinator
Jan 1, 2012 at 1:02 AM

When you use limit=400 (the default in new version 0.6.0 is now 1000), do you get back multiple objects? Also, are you sure your datetime format is correct?

Jan 1, 2012 at 3:13 AM

I tried out a small C# program using the Facebook SDK, but it too ran into the "cannot see any posts prior to November 5, 2010" issue, despite having logic to handle "paging.next" as show in the code snippet below:

                    DateTime oldestCreatedDateTime = DateTime.MaxValue;
                    uint totalPostsFound = 0;
                    dynamic posts = fb.Get("/me/posts?limit=40&since=2009-12-31");
                    bool foundPosts = !string.IsNullOrWhiteSpace(posts.data.ToString());
                    while (foundPosts)
                    {
                        foreach (var post in posts.data)
                        {
                            totalPostsFound++;
                            string postType = post.type;
                            string createdTime = post.created_time;
                            DateTime createdDateTime = DateTime.Parse(createdTime);
                            if (createdDateTime < oldestCreatedDateTime)
                            {
                                oldestCreatedDateTime = createdDateTime;
                            }
                            Debug.WriteLine(createdDateTime);
                            string postId = post.id;
                            Debug.WriteLine(postId);
                        }
                        if (posts.paging == null || posts.paging.next == null)
                        {
                            foundPosts = false;
                        }
                        else
                        {
                            string urlOfNextSet = posts.paging.next.ToString();
                            posts = fb.Get(urlOfNextSet);
                            // next line is not 100% perfect, but works most of the time
                            foundPosts = !string.IsNullOrWhiteSpace(posts.data.ToString());
                        }
                    }
                    Debug.WriteLine(oldestCreatedDateTime);

I get about 327 items total (40 at a time) but it's as if nothing I posted prior to November 5, 2010 is accessible via the Graph API even though I can see everything as far back as early 2009 when I login to Facebook online.  Eventually, I'd like to fb.Delete(postId) on a selective basis, but that's for later.

Coordinator
Jan 1, 2012 at 3:59 AM

Do you get the same "nothing older" effect when you leave out the "since" term? Try that, and try some other formats for the date. See https://developers.facebook.com/docs/reference/api/ which references http://www.php.net/manual/en/datetime.formats.php.

Jan 1, 2012 at 12:05 PM

Removing the &since= and date value returns 327 total posts, the oldest of which was created on 11/5/2010 at 8:44:13 AM

It's becoming more and more apparent to me that Facebook wants to make it as hard as possible to see old information via the Graph API and even even harder to actually delete it through that same API.    I did some research and found numerous other people faced with the same issue and zero solutions that actually work.

As for date formats, I see that ISO 8601 is "the" preferred standard for date formats, which is what I used to specify both since= and until= "experiements" with the Graph API, both of which failed to return any posts older than November 5, 2010. Any examination of the JSON data returned shows how the Facebook Graph API prefers to deal with date/time values.

It looks like I'll be logging a lot of time on my keyboard cleaning up my old posts instead of creating an automated method of doing the same thing.

Coordinator
Jan 1, 2012 at 6:51 PM

That's a shame -- that said, we already knew that even if you could have enumerated the older posts, you couldn't automatically delete them unless they were created by your application.

When I go back through my own https://www.facebook.com/#!/jon.newman1, I don't see any of my own posts older than November 2010, although older posts from other people on my wall are still there through 2008. This is consistent with Get-FBPost; Get-FBFeed only returns 69 items through June 2010.