Brad Ideas - data hosting - Comments

Lots of good work here

brad — Mon, 22 Sep 2008 11:44:06 -0700

But it doesn’t resolve — perhaps nothing can — the almost tautology that making it easier for outside programs to get at the data makes it easier for the data to flow outside. The challenge is how to serve two masters:

Making it easy to program applications that use the data
Keeping the data close to your vest

Not like gears

brad — Thu, 05 Jun 2008 23:23:01 -0700

This is for social apps that interact with your friends, so it won’t be able to do much offline.

okay that makes a lot more sense

gpshead — Thu, 05 Jun 2008 21:54:04 -0700

Similar in some ways to what the Gears browser extensions allow people to do with their web apps... Allows them to run locally including storing data locally when the user is offline and resync with the cloud when back online.

No latency

brad — Sat, 24 May 2008 22:24:39 -0700

You misunderstand. The data is not divided from the app. The apps run on the data host. In fact, if the data host is your pc, you get superb latency.

In that mode it’s a mixture of the old PC model and the new cloud model. Like the cloud but your PC becomes part of it and the browser is your window to the app running on your PC. Sounds silly at first, but it means you can get real portability.

Latency is the name of the elephant

gpshead — Sat, 24 May 2008 12:23:04 -0700

An application that is hosted separate from its data will by the nature of the network have much higher latency and lower bandwidth access to the data.

This limits the user experiences such application can provide.

The right to cache data locally at the application host is a partial work around to this. In reality even the caching has high latency if you want to keep it coherent with other applications views of the same data. Locking the data to be owned by the application to avoid cache sync and verification latencies is effectively the same as not using your own data host in the first place as it essentially gives up having the data host be the master copy of the data.

I expect caching data locally with an application is likely just as expensive to boot strap as providing the data hosting yourself in the first place. caching = hosting.

Regardless, users being able to buy high quality data hosting with APIs such that applications external to the hosts can make great use of them sounds exciting. It leaves users with much more control over their data.

To contact me

brad — Fri, 16 May 2008 20:03:25 -0700

Send me an email!

We Should Talk

Allan Sabo — Fri, 16 May 2008 18:02:02 -0700

Brad,

Read your post at Scoble's blog after posting my own - your must have come in about the same time.

I had to read (ok, quickly scanned) your post here, and find you and I are very much on the same page.

I am working with a group that wants to develop what it believes is the answer - perhaps we should talk.

Please contact me at your convenience,

Allan Sabo
Alti Success Strategies
Experts at Integrating Social Media and Internet Marketing

Data Spaces in the Clouds

Kingsley Idehen — Thu, 08 May 2008 13:55:39 -0700

Brad,

What you describe is what I've referred to as Data Spaces in the Clouds (Fourth Platform) for a while :-)

Yes, there is some confusion about the literal interpretation of the phrase: Data Portability (free movement of data across platforms). We don't necessarily want free movement of data across realms (unless we explicity enable it in our space). Instead, I believe we seek Open Access to our Data Spaces with access control granularity.

To conclued, we do need Data Access by Reference facilitated by portable Data Containers (Data Spaces) in the Clouds. Of course, these containers can move themselves, or data from the clouds to other locations, wholesale or via replication and synchronixation. In all cases using standard protocols and existing infrastructure such as the Internet and Web.

Links:

1. OpenLink Data Space Wikipedia Page
2. OpenLink Data Spaces (Open Source Edition) Home Page
3. My Data Space Profile Page
4. EC2 installation Guide
5. How to get a Gateway into your Data Space (i.e a URI for Your Data Space) in 5 minutes or less
6. Recent WWW2008 Presentation about Data Portability and Data Accessibility (PPT)

FoF apps

brad — Wed, 07 May 2008 11:26:57 -0700

As I’ve noted, FoF apps turn out to be much less interesting than people thought at first. Do you really look at the photos of your FoFs? The main FoF app that seems to be useful is LinkedIn’s “search your network” which can answer questions like, “Who can I contact at Company X” and dating introductions. FoFoF turns out to be surprisingly non-useful.

However, I won’t proclaim that nobody can think of useful apps (or simply entertaining) apps here. So there need to be solutions, even if it turns out to be that those apps get access to large networks, but are the only ones that do. (Remember, our alternative today is zillions of apps getting access to this data.)

I don’t expect home PCs to be required here. Everybody wants an always on host for many functions. You don’t need it in the sense that I don’t think “Ask 100 hosts to search for a query” is a good implementation, but I would use the local hosts just as a way to do things efficiently for the user when the user is signed on. The cloud host would do things for others. Client data hosts and cloud data hosts would sync.

Your particular app, and other FoF apps, could be implemented, somewhat less efficiently, with data updates. That is to say, if you are using an FoF app, you would send changes to your friends, and they would forward those changes on to their friends as part of the update stream. In this case everybody is storing all the basic data (not big things like photos, just smaller stuff, including the URLs/access tokens of the photos) on their own host, and apps can operate on it. This is why it does not scale to FoFoF, but I think it could handle FoF if the updates are not large.

Of course, you, your Fs and your FoFs must be running the same application, but that is the same as saying they are all members of Flickr. To implement Flickr you need more though, and it may not even be possible to implement all apps in this manner. Still better than implementing them all in a central repository manner, though.

Interesting approach

brad — Wed, 07 May 2008 11:16:17 -0700

But I haven’t studied the underlying systems proposed enough to judge if they can do it. One concern I see immediately regards whether developers can be talked into it. Part of the sex appeal of web 2.0 (meaning apps in the cloud) is that developers get free reign to write and maintain their apps using whatever platforms and tools they like. They are no longer limited to even the constraints and problems of writing code for a user’s PC. Users at the same time love not having to install software, having somebody else maintain it all, and having to roam.

My own proposals face this problem too. These abilities are very attractive to users and developers, and as long as they can get the functionality (which javascript has now given) they will rush to them.

It is for this reason that I have decided that some compromises will be needed, that we won’t get to the level where we can run a malicious app on our data. That’s because the programming hoops required to use a system that bars malicious apps may be too involved. Happy to be proven wrong, though. I would be happy just to reach the level where apps don’t end up taking more data than they need, and don’t end up storing copies of it.

W5 Project at MIT

Evan Jones — Wed, 07 May 2008 04:51:52 -0700

The W5 project at MIT is looking at ways to solve these issues:

http://pdos.csail.mit.edu/~max/docs/w5.pdf

Searches

Radovan Semancik — Wed, 07 May 2008 01:14:33 -0700

I really wonder how you can efficiently implement this use case:

"Show me the flickr photos that my friends and the friends of my friends faved in last 2 weeks, sorted by the total number of favs", while the photos are still stored on flickr.

I can see you can implemente that with "agent" that will crawl to the hosting sites of your friends and their friends, collect the data and come back. But that would be slow, especially if some of your friends host their data in PCs that are currently offline (remember The Eight Fallacies of Distributed Computing?). Do you have any solution for that in mind?

BTW, creating a Virtual Machine you still provide API to the applications. But it is quite broad, which means it is difficult to control security.

Finding the host

brad — Tue, 06 May 2008 14:48:00 -0700

Well, in the past we have used DNS as a way to name hosts and move them around. This could be used here, but not a high level domain. I might get bradsdata.datadns.com as a subdomain that I can point at whatever data host I like. However, there could be other indirection or discovery protocols.

Typically I see a main site providing your interface to social data apps. That site would embed other web pages which are served from your data host, running code provided by app providers. The DNS would direct your own browser at your own data host of choice. (In fact, this architecture allows the data host to be your own PC, if you don’t need to roam. Your own pc at localhost:port would see a request for a social app window. The data host program on your PC would connect to the specified remote application’s server for any code updates or special data, download them if it doesn’t have them cached, execute the code and return the results to the iframe in the browser page.)

I think I’ll update about that.

Now as for the central repository: This is complex. People are saying, it seems, that they don’t want their data scattered around everywhere, both because of lack of control, but more commonly because the UI to give apps access to it is too complex. If we can develop a good UI so that it is easy to give apps just the data they need, and no more, then scattering can be good. The data hosting model does not dictate about scattering or centralization, but I agree that users will tend to centralize, just for ease of control. A central server contracted to me may be better than 30 servers with only loose bonds to me, such as 30 different social app companies each knowing different subsets of my data.

Service portability good. Centralized servers not necessary.

Joe Andrieu — Tue, 06 May 2008 12:26:43 -0700

Good post.

It's clear that the data portability model is limited, but I would even go beyond hosting portability to service portability, and I would separate the hosting company from the value-added service provider.

First, the hosting model still leaves you at the mercy of the host if/when you decide to move... you still have to update your address at all of the service providers who might access the services of your hosting company. A more robust model would provide service portability through service discovery. Service providers don't need to know where it the data is hosted, but rather where they can find the current host. That gives you a layer of indirection that lets you move your hosting company without needing to remember which service providers are currently relying on that host.

Second, you suggest that the hosting company's job is to perform actions on your data.

Why is that?

My DNS host doesn't perform functions on my DNS. Nor do I expect my webhost to perform actions on my website, although I do like to have a range of services I can easily install and run (such as installing WordPress through Fantastico).

I would argue that there is an inherent conflict of interest in the hosting company providing value-added services, and that in fact, what we should do is design an architecture where hosting is functionally distinct from value-added services. Any authorized value-add service provider should be able to access your data services, which leads to a cleaner architecture where companies that happen to provide both hosting and value-add services can do so with clear contracts and authorization.

Finally, there's no real need to have all of my services at the same hosting company, just like my DNS, my website, and my email can all painlessly be hosted anywhere I like. As long as the services can be discovered, there's no reason to have any individual's services centralized, nor a need to centralize many individuals' data in one place.

The collaborative/co-op type negotiating strategy you suggest can easily be implemented by a value-add service provider, independent of the source of the data. Users simply join that co-op or buying group and point the co-op to their discovery service. Flash-mobs rejoice.

All of which is to say, you are definitely going down the right path. More portability more better.

-j

It matters where it is hosted

brad — Tue, 06 May 2008 11:42:59 -0700

Because we want to lay down the duties of the people who host it for us, if we don’t host it ourselves. (And most people won’t host it themselves.)

If a host has just one duty, to keep the data safe and allow only authorized actions on it, then that is what the host will focus on. If the host is facebook, whose duty is to find ways to monetize its database to maximize shareholder value, you will get a different result.

That’s why the layers, and why for most people, not having too many hosts. (Some might decide to have multiple hosts to maintain multiple persona, such as a personal and work persona, but that’s their decision.)