base2smug (pbase to smugmug gallery convert) I'm not so sure I'm saving labor anymore

crenner

OK. I got my API keys.

I started coding (yesterday) a VB6 program to convert pbase galleries to smugmug galleries (it's called basetosmug). I still use VB6 a lot for it's huge compatibility. It runs on every Win32 and Win64 platform at this time.

I thought it would be some short weekend work (hah!). Just parse all my pbase galleries for all the needed elements in the HTML, and then use the smugmug API to upload the equivalents.

This code is going to take me much longer than I first imagined. While MSXML is pretty simple to use, there is still a learning curve for me, as I have never parsed XML before. The API documentation is, also, (how should I say it), umm, incomplete. Nothing that cannot be figured out by a good programmer/debugger, but still incomplete.

I've skipped all the pbase parsing for now (as I have already done similar work in PHP scripts for one of my sites), and went straight to tackling the smugmug API. I ended up using REST (it seemed easier to figure out than XML-RPC, but it was about a 80% arbitrary decision because I had to start somewhere). I'm using good OOP coding conventions, and all my API code for smugmug is in a single class-module (so I can compile it into a separate DLL from my EXE).

I now have a simple form that can login to the sm API (create a SessionID), get Albums, get Categories, and create a minimal album (title and category only for now), and logout.

This (so far) has required the following extensions to VB6:
* WinHTTPRequst - ditched in favor of MSXML
* MSXML (for making HTTP requests and for parsing XML returned by these requests)
* File Scripting Object (for simplified file operations)
* DAO (for storing and passing data from the API DLL to my form) (I still use MSJET/Access databases for small data sets, as DAO is easier and faster to code for me than ADO)

I have forgone almost all error trapping (for now) as I don't know if I will ever put forth the extra effort to make this a distributable application. Logging is also not present (unless you count Debug.Print as logging)

Done:
1) smugmug API authentication
2) smugmug category downloading, parsing, and database dumping
3) simple smugmug album creation
4) smugmug logout

Still to go:
1) Advanced Album Creation Options
2) Downloading HTML from pbase galleries
3) Parsing HTML from pbase galleries
4) Dumping parsed pbase HTML elements and image links into a database.
5) Downloading "original" images from pbase galleries into a holding location
6) Uploading images into newly created galleries
7) Uploading parsed pbase data elements into galleries with images
8) Error trapping (only if I am going to distribute this)
9) Logging (only if I am going to distribute this)
10) Progress indicators (only if I am going to distribute this)
11) Installer (only if I am going to distribute this)

I'm now on the third day of my smugmug trial (14 total days). I no longer believe that I can have this program working by the end of the weekend. I'm hoping that I can have this program working (if not pretty), by the end of my trial subscription to smugmug. It doesn't seem likey, considering meeting this deadline would probably mean neglecting my daughter, wife, and job.

So far, 100% of the code for this program is mine. I have avoided the temptation of using available source (like that of the VS 2003/C# program "Send to SmugMug"), as I am still undecided about ever distributing (for free, donations, profit, or otherwise) this application. So, in order to avoid possible future distribution problems, I've written completely my own code.

If anyone has already created a class module/DLL of the SmugMug API, usable by VB6, and is willing to let me use it without many restrictions, than I would be appreciative. Otherwise, I'll keep coding away.

Feel free to provide comments or feedback on any of my inane babble above.

Andy

Sent you an email

crenner

Progress Report on basetosmug tool
Andy, I did get your email. Thank-you very much! (See my email response)

I'm making headway.

Here is a progress report on basetosmug:

Done:
1) smugmug API authentication
2) smugmug category downloading, parsing, and database dumping
3) simple smugmug album creation
4) smugmug logout
5) Downloading HTML from pbase galleries
6) Parsing HTML from pbase galleries (works for me, but not tested on a wide variety of galleries at this point)
7) Dumping parsed pbase HTML elements and image links into a database.
8) Downloading "original" images from pbase galleries into a holding location

Still to go:
1) Advanced Album Creation Options (Description has been added only since my last report)
2) Uploading images into newly created galleries
3) Uploading parsed pbase data elements into galleries with images
4) Error trapping (I've added just some minimal error trapping for my own sanity)
5) Logging (only if I am going to distribute this)
6) Progress indicators (only if I am going to distribute this)
7) Installer (only if I am going to distribute this)

crenner

basetosmug WORKING!
basetosmug is NOW WORKING!!! The GUI is ugly, it's undocumented, it's easy to produce a fatal error, but it does work!

I'm actually going to have to start assigning version numbers soon!

Here is a progress report:

Done:
1) smugmug API authentication
2) smugmug category downloading, parsing, and database dumping
3) simple smugmug album creation
4) smugmug logout
5) Downloading HTML from pbase galleries
6) Parsing HTML from pbase galleries (works for me, but not tested on a wide variety of galleries at this point)
7) Dumping parsed pbase HTML elements and image links into a database.
8) Downloading "original" images from pbase galleries into a holding location
9) Uploading images into newly created galleries
10) Uploading parsed pbase data elements into galleries with images

Still to go:
1) Need Bug-Fix: Position order is EXACTLY backwards. I need to upload the images in reverse order to fix this, or learn how to assign an position number at upload time.
2) Need to verify if FileNames are/were preserved by uploading a gallery without captions
3) Need to verify if EXIF information is/was preserved
2) Advanced Album Creation Options
3) Error trapping
4) Logging
5) Progress indicators
6) General clean-up
7) Installer

Example of a converted gallery using basetosmug:
http://www.pbase.com/crenner/nicole_grace_renner_jan_2006 (original pbase gallery)
http://crenner.smugmug.com/gallery/1854903 (converted smugmug album)

Notice that the captioning, gallery title, and gallery description were preserved and converted along with the images!

Andy

crenner wrote:

basetosmug is NOW WORKING!!! The GUI is ugly, it's undocumented, it's easy to produce a fatal error, but it does work!

It's been great to watch your progress Keep up the wonderful work.

ivar

crenner wrote:

basetosmug is NOW WORKING!!!

clap

samilton

crenner,
I spent some time working on a pbase to smugmug tool. While I never got it finished I thought I would give you a few pointers that I ran into. Hope you don't mind them...

1. I tried testing against two or three pbase galleries and found that all of them had issues with the validity of the HTML. I ended up having to put the HTML through a html tidy api.

2. It might have been the photo's in specific galleries, but it seems to me that the PBase image resizing processes removes the EXIF. Stick to using the original.

3. Finally, it was next to impossible to work with protected galleries. I would suggest you not go down that path and suggest to anyone converting a PBase gallery to Smugmug that they just unprotect the gallery in question during the conversition process.

Hope this helps.

Andy

samilton wrote:

crenner,
I spent some time working on a pbase to smugmug tool. While I never got it finished

Slacker!

So, crenner, if you want the source from Sam's effort, I can get that for you - just holler by email.

crenner

Thanks!

samilton wrote:

1. I tried testing against two or three pbase galleries and found that all of them had issues with the validity of the HTML. I ended up having to put the HTML through a html tidy api.

My pbase parser is a hack. I'm scanning for expected strings, and not parsing the HTML at all. I've been using similar code to parse my pbase galleries of my daughter and semi-automatically create the pages on the following site:
http://nicole.rennerfamily.org for about a year-and-a-half.

samilton wrote:

2. It might have been the photo's in specific galleries, but it seems to me that the PBase image resizing processes removes the EXIF. Stick to using the original.

I did anticipate this as a problem, but the current version of the code does nothing to address this at all. As I am working on a version for distribution, I will keep this in mind.

samilton wrote:

3. Finally, it was next to impossible to work with protected galleries. I would suggest you not go down that path and suggest to anyone converting a PBase gallery to Smugmug that they just unprotect the gallery in question during the conversition process.

I'm being selfish on this, but that was my plan. I have either little or no protected content on pbase, and really don't want to address authentication, sessions, cookies, or anything else on pbase.

Thanks for the feedback!

crenner

follow-up
This is just a brief follow-up to this thread to note that basetosmug alpha versions have been released. (it hadn't been mentioned in this thread before)

sshaw

crenner wrote:

This is just a brief follow-up to this thread to note that basetosmug alpha versions have been released. (it hadn't been mentioned in this thread before)

I am having difficulties getting past the following error when using your latest build for versoin 0.2. Pbase.com logs on fine, as does "getting Pbase.com Galleries" and getting pbase.com Images. I then:
- Click on logon to Smugmug. No response indicates that it is successful.
- If I assum it is successful, I click on "Create Smugmug.com albumns".
- The error message I get is:

Unexpected error:
Error number: 91
Error description: object variable or block variable not set.

I've been wanting to migrate to Smugmug for months, but have held back due to the massive task of nigrating approx, 1800 photos, with extensive captions and protected galleries. If there was an easy migration facility to do this, I think youl would find a mass exodus over to Smugmnug. The problem has become more acute now, with the loss of a significant number of photos due to a recent RAID array failure.

I am looking forward to your response, and, perhaps a 'roadmap' on how to best acheive a successful migration from someone who has done this successfully.

Thanks,

Stan

crenner

I can help you (sorry for the slow response)
Stan -

Sorry for the slow response. I didn't know anyone was using this project anymore. It's been inactive for months. I can help you with this conversion. The error you got was most likely due to a missing package or ActiveX control.

Please make sure you have full debugging on, and send me the resulting log file. I can probably trace the problem. I also can setup a gotomeeting.com meeting or something to see for myself.

I need to dig up the code. I haven't touched this project for almost a year.

I'll send you a IM with my email address.

sshaw wrote:

I am having difficulties getting past the following error when using your latest build for versoin 0.2. Pbase.com logs on fine, as does "getting Pbase.com Galleries" and getting pbase.com Images. I then:
- Click on logon to Smugmug. No response indicates that it is successful.
- If I assum it is successful, I click on "Create Smugmug.com albumns".
- The error message I get is:

Unexpected error:
Error number: 91
Error description: object variable or block variable not set.

I've been wanting to migrate to Smugmug for months, but have held back due to the massive task of nigrating approx, 1800 photos, with extensive captions and protected galleries. If there was an easy migration facility to do this, I think youl would find a mass exodus over to Smugmnug. The problem has become more acute now, with the loss of a significant number of photos due to a recent RAID array failure.

I am looking forward to your response, and, perhaps a 'roadmap' on how to best acheive a successful migration from someone who has done this successfully.

Thanks,

Stan

sshaw

Thanks very much for your response. I agree, from the look of the error messages, it does look like an ActiveX control or package may be missing. I'll send you my debug log file in a few minutes. If we can resolve this, I will be delighted to report success on this forum.

Cheers,

Stan

crenner wrote:

Stan -

Sorry for the slow response. I didn't know anyone was using this project anymore. It's been inactive for months. I can help you with this conversion. The error you got was most likely due to a missing package or ActiveX control.

Please make sure you have full debugging on, and send me the resulting log file. I can probably trace the problem. I also can setup a gotomeeting.com meeting or something to see for myself.

I need to dig up the code. I haven't touched this project for almost a year.

I'll send you a IM with my email address.

TheRedDragon

pbase to Smugmug
can you use this program on a MAC ???

TheRedDragon

error message - pbase to smugmug
Unexpected error:
Error number: 91
Error description: object variable or block variable not set.

along with numerous other error numbers. i tried it on a PC.

ifokus

base2mug
I have tried the base2smugmug several times after downoading, and I get all kinds of errors. Is there any other way to get all my pbase photos over to smugmug ?

Thanks!

TheRedDragon

pbase to smugmug
i do not believe so. I searched around but only found this program and it doesn't seem to be working. all i get are error messages also.

sshaw

Could SmugMug possibly help to develop a reliable PBase import utility?
About a month ago, I received a very nice email from "crenner", the fellow who wrote the original utility. Apparantly, it worked beautifully last year, but something has changed (possibly within PBase) which broke the utility. The problem is that he is very busy these days and was essentially volunterring his time to support a home-made utility originally designed to move to move his files to SmugMug.

I can completely understand, given SmugMug's great features, why crenner would not need to enhance/fix this utility now. One of the big 'enhancements' needed is to support private galleries. Many Pbase users have them, but even if the utility worked in its present state, we would have to remove any password protection in galleries before running the conversion utility. I wonder how many people would want to do that? The problem is crenner has little need to either fix the utility, or to enhance it. He already has his photos in SmugMug. The only gift or payment he would likely receive is our heartfelt thanks.

Which brings me to my question. Is it not in SmugMug's best interests to develop and support a solution that will actually help import these photos into this commercial application? Sure, a 3rd party enthusiast could write one, but the real beneficiary is SmugMug, and, of course, PBase users who dread moving literally thousands of photos manually (I have over 1800 heavily annotated photos, in 106 public and private galleries !! ). I can only imagine the short-term and long-term financial benefit building a utility that is supportable on an ongoing basis. Crenner showed the way... but what we really need is corporate support behind this effort.

So, to my friends at Smugmug, can you possibly help with this?

Best regards,

Stan

TheRedDragon wrote:

(In reference to the current PBase import utility, "base2smug"): i do not believe so. I searched around but only found this program and it doesn't seem to be working. all i get are error messages also.

TheRedDragon

Pbase to Smugmug
I wish I had the smarts to build a program.. but I wouldn't know where to begin. . . to bad it doesn't work anymore.

devbobo

Stan,

Pbase is on my list of things to add to SmuggLr. Unfortunately, due to lack of API access or sufficent feeds on pbase, the only way to do it is with scraping the pages.

Cheers,

David

sshaw

Hi David,

Thanks for your quick response. I am absolutely delighted to hear you are planning to support it soon!

Here is some software that might help, a backup utility called pbgrab. It was discussed in an end-user's support forum, at http://forum.pbase.com/viewtopic.php?p=225717 . The latest utility is located at:

http://srijith.net/pub/pbgrab_latest.zip


If Smugmug was able to run this conversion utility (or something like it) on a SmugMug server, bandwidth and latency problems dealing with a local end-user connections could be avoided. From there, it might be possible to design a custom SmugMug import facility to move files from the temporary holding located on the SmugMug server, into a permanent location, and reassign comments fields as required. It's a bit of a kludge, but it might work.


Best regards,

Stan

devbobo wrote:

Stan,

Pbase is on my list of things to add to SmuggLr. Unfortunately, due to lack of API access or sufficent feeds on pbase, the only way to do it is with scraping the pages.

Cheers,

David

devbobo

sshaw wrote:

Hi David,

Thanks for your quick response. I am absolutely delighted to hear you are planning to support it soon!

Here is some software that might help, a backup utility called pbgrab. It was discussed in an end-user's support forum, at http://forum.pbase.com/viewtopic.php?p=225717 . The latest utility is located at:

http://srijith.net/pub/pbgrab_latest.zip


If Smugmug was able to run this conversion utility (or something like it) on a SmugMug server, bandwidth and latency problems dealing with a local end-user connections could be avoided. From there, it might be possible to design a custom SmugMug import facility to move files from the temporary holding located on the SmugMug server, into a permanent location, and reassign comments fields as required. It's a bit of a kludge, but it might work.


Best regards,

Stan

Hey Stan,

SmuggLr uses the SmugMug's api, specifically smugmug.images.uploadFromURL. This method basically allows the SmugMug servers slurp the image directly from the URL provided, so there is no need to download the file.

Not sure how soon it will be, but it's on my list

Cheers,

David

TheRedDragon

pbase to smugmug tool
I can't express how badly i need and want this tool. i have two pbase accounts with about 7000 photos I would like to transfer to smugmug... i have my fingers crossed for an application soon.

JLNJLN

Damned !!! I've been trying basetosmug all week long too !
Like sshaw, i've been trying to use basetosmug and got the same exact result ...
Too bad I was really hoping to get my galleries into smugmug.

I understand very well the fact that crenner is not really motivated to invest on ihis application anymore ...

I may try a 2 step approah, using one of the "Pbase to file" application, and then upload again from file ...

Not too excited about it ...

TheRedDragon

Pbase to Smugmug
let me know how you make out. i still have 6000 photos to transfer.

sshaw

Hi, just checking in. Is there any progress on this? I'm not surprised to see others in a similar quandray. I realize that PBase is in the dark ages by not publishing an API, but there simply must be a way.

Andy

sshaw wrote:

I realize that PBase is in the dark ages by not publishing an API, but there simply must be a way.

All if you know of one?

sshaw

Andy,

I'm not a programmer, however, here is one possibile approach. There is a very functional utility developed for Pbase that allows users to download photos as a backup, called Pbgrab. It seems to work well, preserving captions, comments., etc. It also supports access to private galleries, so they can be downloaded to a hard disk without having to expose photos to public access, as did basetosmug, when it was working. The utility can be found at:

http://pbasewiki.srijith.net/tools

The problem with Pbgrab is that it only downloads files to a local hard disk. To bring files into Smugmug, you would still need to recreate the heirarchical structure, upload photos manually into SmugMug, cut/paste comments and captions, etc. That would be totally impractical for thougsands of photos.

However, I would imagine a similar approach to the way Pbgrab was developed (whatever it is, programatically) could be used to create an efficient PBase to Smugmug utility. One suggestion I would make is that to make efficient use of network bandwidth, it would probably be better to set up the utility as a hosted middleware application on Smugmug's servers. That way, end users would not have to download images files to temporary directories on their computers, then transfere them back to SmugMug.

Regardless, I suspect the amount of work involved in creating such a utility would be more than compensated by hundreds of Pbase users who would be delighted to use this facility once word got out that it exists. I look forward to your further thoughts on this.

Best regards,

Stan

Andy wrote:

All if you know of one?

thebusch

Any updates on this project/software?
I could certainly use this transfer utility and I'm sure MANY others could also use it! It looks like a gold mine waiting to be tapped.

Andy

thebusch wrote:

I could certainly use this transfer utility and I'm sure MANY others could also use it! It looks like a gold mine waiting to be tapped.

The update is we need a 3rd party programmer volunteer to take this on, any interest?

thebusch

Andy wrote:

The update is we need a 3rd party programmer volunteer to take this on, any interest?

Hi, Andy! Well, don't look at me, I can turn on a computer but that's about all. Guess I'll start uploading a few at a time and keep hoping Santa will bring us something to make the job easier.