Beta Release: DotNetNuke Amazon S3 Folder Integration Providers
I am pleased to announce a beta release of my Amazon S3 integration authorization and data providers. It may be downloaded via CodePlex on its project homepage. As is all of my DotNetNuke work, this project is fully open-source and available under a liberal BSD license.
The DotNetNuke web application framework offers multiple file persistence options out-of-the-box, including file-system storage (both unsecured and secured by ACL), along with ACL-secured database storage. When creating a link to a resource, the DotNetNuke UI provides a convenient list of these files, and also allows direct input of arbitrary URI.
However, there exists no ready method by which an administrator might link to a known set of files persisted external to the installation. While direct URI input might be used here, it requires knowledge of these data, and does not allow for enumeration and management of the external objects themselves.
This project attempts to bridge that gap by integrating resources persisted on the Amazon S3 into the DotNetNuke framework. Resources stored there are enumerable via the File Manager and selectable via the URL control. Throughout the core framework, these external resources are treated identically to database-secured resources, including observance of Amazon S3 ACL, automatic synchronization, and (reasonably friendly) 301 Redirects to the Amazon S3 when accessed via LinkClick.aspx.
This is effectuated via customization of two providers: authorization and data. The authorization provider integrates Amazon S3 ACLs for external resources, and the data provider allows enumeration of and details about the external resources themselves.
Project Goals
- Integration with the Amazon S3 for data storage external to the DotNetNuke framework.
- Seamless integration into the DotNetNuke UI, both at the File Manager and URL controls. For example:
- Amazon S3 objects should appear in the File Manager just like any other internal files,
- Amazon S3 objects should appear in the core URL control, and
- Insofar as is feasible, Amazon S3 files should be indistinguishable from their secured storage counterparts.
- No external Amazon S3 UI or kludgy link generation
- Whenever possible, utilization of 301 Redirects to Amazon S3 resource URIs.
- Full observance of the external Amazon S3 bucket ACLs.
- Require no core changes and utilize only existing DotNetNuke extension points
Configuration and Usage
After installation, two new profile properties will be automatically created across all portals:
- S3Key, and
- S3Secret
The names of these properties may be changed in the web.config near the configuration/dotnetnuke/permissions node). To enable a portal for Amazon S3 integration, the key and secret for the portal administrator must be configured with valid values.
Any user may submit his or her own key and secret information at the profile level; these data will be used to circumscribe the set of available Amazon S3 objects for that user.
After a portal’s administrator account is configured with a valid Amazon S3 key and secret, the file manager will display those buckets and objects stored on the service:
A screenshot demonstrating Amazon S3 cloud-based integration on the DotNetNuke platform
The list of available buckets will be circumscribed by the permissions granted by the administrator’s S3 account. Because DotNetNuke does not support file-level ACLs, objects within an Amazon S3 bucket are assumed to have an ACL identical to that present on the bucket itself (though a more restrictive policy will still be enforced by the S3 service).
Similarly, the DotNetNuke URL control will display a list of buckets accessible by the accessing user, and allow selection of the objects therein:
Amazon S3 Demonstration Displaying URI Selection
Discussion
Overall Authorization Model
There are a number of areas in which the authorization policy enforced by the provider might be improved. In its current form it is minimally complete, but could easily be made more robust and sensitive to user context (in particular, no attempt is made to deal with Amazon S3 user-specific ACL assignment). The provider is largely aimed at objects intended to be made available to “everyone.”
File-based ACLs
DotNetNuke does not support file-based ACLs, but these are allowed and enforced by the Amazon S3 service.
Generated URIs
The 301 redirect URIs created by the data provider are not per-user, and thereby require read-permission on the “everyone” group for the S3 object in question. A stronger model might generate a valid read URI for the accessing user, so long as that user has read permissions on the object itself.
CRUD
Only read access is implemented for Amazon S3 buckets and objects; while it remains quite straightforward to implement the other access characteristics, these are left to the (many) other S3 managers that exist for these purposes.
Shared secret storage
Shared secrets are stored in plain-text on a user’s profile; this is generally a horrible security policy (Alex Shirley will certainly yell at me for doing this after I strongly discouraged him from doing so in a similar project). Because federated authentication is not possible with the Amazon S3 service, this secret cannot be made application-specific and must be stored somewhere. The solution here is an encrypted profile property type; this is clearly outside the scope of this project, but is on my radar for a future complimentary project.
Performance
A number of internal operations are O(n*m). While some of these are cached by the underlying DotNetNuke framework, the integration implementation itself could be improved via additional caching and lookup tables.
Your Feedback is Needed!
As this project moves toward an initial production release, it will rely heavily on the feedback provided by the community. Any such feedback — including comments, ratings, and constructive criticism — is greatly appreciated.
B
From the Desk of Brandon Haynes » Presentation Materials: Advanced Authorization in DotNetNuke 5.1
June 22, 2009 @ 3:17 pm
[…] are displayed right). The code for this demonstration will be made available later this month on its project homepage at […]
Alex Shirley
July 25, 2009 @ 4:33 pm
> Alex Shirley will certainly yell at me for doing this
Yell! Yell! Yell!
– of course you were right :).
Jonathan
August 3, 2009 @ 2:03 pm
Hi Brandon — Thanks for creating this. I’m playing around with it and it appears to work well.
Can you clarify whether or not changing the data provider to your provider conflicts with your goal of not changing the “core”?
Thanks.
Jonathan
August 3, 2009 @ 5:33 pm
The files are corrupt when I try to open them from either DNN File Manager or the Links module. They were uploaded with CloudBerry.
Any ideas?
Thanks!
Tozzi
August 11, 2009 @ 3:04 pm
Excellent extension, a great idea to leverage the cloud storage especially for high volume sites. My ISP hosts a DNN site that pushes in excess of 80 GIGs a month and storing this in the cloud will be awesome!
One grip, the extension requires DNN 5.1+ and we’re running over 100 sites on DNN 4.x.x which will not be upgraded any time soon. How much effort to make this backward compatible with the 4.x branch?
Brandon Haynes
August 11, 2009 @ 3:40 pm
@Jonathan: The DotNetNuke providers are one of the (many) valid methods by which one may safely extend the framework; this extension is, therefore, appropriate (and does not modify the core in any way). One could, however, make a reasonable argument that the data provider is the “least safe” such provider. C’est la vie.
With respect file corruption, I have seen circumstances where AmazonS3 will report an error directly in the file stream (and the resultant attempt to interpret the data as, say, a JPEG will appear as corruption). Try persisting the file to disk and examining it with a text editor. Sometimes the permission problem will become evident in the text.
@Tozzi: The extension is complied against the 5.1 version, but does (to my recollection) not utilize any functionality that is not available in the 4.x line. A recompilation against 4.9.4 should be fairly straightforward, though there will be some small API changes that are necessary due to method obsolescence.
B
Karl
August 31, 2009 @ 10:49 pm
Everything works except for clicking on the link in a HTML/Text box. When configuring a link on an HTML/Text box, I can see the directories and select a file. When I click on the link, the web page cannot be opened. Using Cloudberry Explorer, I’ve tried to give every ACL group and user read permission and I even created a user call everyone and I gave it read permission.
Any ideas?
Brandon Haynes
September 11, 2009 @ 8:15 am
@Karl I’ll look at attempting to reproduce your issue. In the meantime, I suggest using a debugging proxy (such as Fiddler) to see where the link handler is attempting to redirect to; that will yield insight as to whether the problem is with the provider or is an S3 configuration issue.
If you could open a work item on the project website outlining this issue, it would be greatly appreciated.
B
Adam Jones
September 23, 2009 @ 2:11 am
I have over 8,000 photos listed:
http://yokosukabase.com/pics.aspx
I think I need S3 amazon integration.. It would help with performance, and for storage space. Thanks!
I am running 4.9.5 and dont plan to migrate to 5 for a few more months. If there is a way, that you can make this API available on the 4 release, that would be fantastic. I wouldnt mind testing this capability, and seeing if I can get things working..
Brandon Haynes
September 23, 2009 @ 8:09 am
Hi Adam,
The S3 provider utilizes the authorization architecture introduced in DotNetNuke 5.1. As such, it is not available for extension in the 4.x line and S3 integration is therefore not possible in this version without (significant) core modification. It is likely that an upgrade to 5.x (in particular, 5.1.2 is a highly stable choice) will be your sole recourse for S3 functionality.
B
Will Sugg
December 14, 2009 @ 4:54 pm
Hi Brandon,
I am in a bit of a quandry. I really need s3 for a DNN site that is completely built around a module that can only be stable at 4.9.5.
I have a dedicated server with Jungle Disk on it which lets you have “local drive mapping”.
Do you have any guidance or ideas on how I can get all the photos and files people are uploading off of my server and onto the Amazon Jungle Disk?
thanks,
Will
Jonathan Puddle
January 11, 2010 @ 7:43 am
Hi Brandon, I sent you an email but have not yet an a response from you. I wonder if you’d be interested in turning this module into a full-fledged CDN extension, so an admin could configure it to be used with any number of (preconfigured) CDNs. We need this functionality in our DNN sites, and would like to talk about sponsoring your development of this if you are interested.
Brandon Haynes
May 15, 2010 @ 11:00 am
Hi Jonathan,
I must have overlooked your e-mail. Try contacting me again!
B
Mark Saunders
July 8, 2010 @ 8:30 am
I am trying to find a way to contact you pertaining to this project. I am having trouble finding your email address or any contact us form.
Mark
August 18, 2010 @ 8:27 am
Have you tried the provider out with any Gallery modules? I assume the core gallery will work with this, saving all files in S3? Has anyone tried Ventrian’s Simple Gallery with this (I plan on trying this week).
Brandon Haynes
August 18, 2010 @ 10:56 am
Hi Mark,
So long as the extension uses the core file system API, there should be little issue with interoperability. I have heard reports that the Repository module does not do so (and thereby would exhibit issues), but I have no information regarding either the core Gallery or Ventrian products.
If you are investigating an extension for potential compatibility, the ability to select, upload, and display files from a “secured” DotNetNuke folder would be a strong indicator that the extension relies on the core file system API, and thereby would be compatible.
Brandon
Brandon Haynes
August 18, 2010 @ 11:02 am
Hi Mark (Saunders),
I am reachable via b haynes at fas dot harvard edu.
Brandon