You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hm ... Let's put this as an enhancement. However - I do not see this in the near future 😁 ... Maybe you want to step in - or know somebody who wants so 😅
We have some external storage backends in Nextcloud, (Amazon) S3 seems to be finished for (primary) storage for Nextcloud 11: #768
This may be an alternative ... Sharepoint is also on the roadmap for 11, if you rely on Microsoft 😁
I find it rather puzzling that blob is not supported.
If you look at the gartner quadrant then there is Amazon and Microsoft as the big players.
Supporting only one of them cuts down potential user base by approximately 50%. Take into account how many personal computers actually run on Microsoft it's even more puzzling to me why Microsoft blob is not on the roadmap. But hey its just my thoughts. Thanks for all the hard work you put into next cloud. All you guys are awesome!
Would love to see this also. May be worth mentioning that (and I haven't tested this) but Azure File Store allows SMB access, so you could potentially run Nextcloud on an Azure VM with external storage mapped to the Azure File Store, as long as the VM and storage were in the same region (otherwise you need SMB3 encryption support, which isn't there yet in Linux). Something I will have to investigate.
Reacted by Marius Blüm, David Ziswiler, M4xAmmo and Bruce Truth
I have a few projects with Microsoft azure for processing and nextcloud for storage and sync right now and with their new push for openness on azure I think there is a good chance their evangelists might be interested in helping to implement that feature.
I'll mention it in the next meeting.
We're also interested in this ability. If the SMB mapping on the host box works would it in theory at least be able to read/write to the blob? If I have to get a blob mounted on the VM and it works well I don't think that would be a very hard work around for the time being.
I've been tinkering with this, using the S3 backend as a template. It's a bit of a work in progress. However, as a temporary measure you can use a Windows server (2008 or greater), with WebDAV authoring role for IIS and essentially act a proxy in between your users and azure blob storage, see this how-to: http://fabriccontroller.net/mounting-your-azure-file-shares-on-premises-through-webdav/ Edit: Actually, it looks like that only supports the Azure Files which is much more expensive than blob.
Should it support serving from the verizon/akamai CDN if it is linked to that storage account?
Also, either nextcloud will need dedicated access to the blob container, or maybe we can use a Service Bus subscription and just mandate that anyone adding/changing/removing a file triggers a message sent, that way we can confidently cache the metadata/contents and there wouldnt be polling which increases cost a lot.
For those who are curious, there are ways to mount Azure Blob storage directly on your VM or in k8s. That's what I ended up doing. Honestly if you're going to use it as primary storage, this is a better option than implementing in the application layer IMO.
Dysk project "Attach Azure disks in < 1 second. Attach as many as you want. Attach them where ever you want. dysk mounts Azure disks as Linux block devices directly on VMs without dependency on the host."
also instead of BlobFUSE you could potentially map Azure Files with SMB directoy to the host vm.
But as you can mount it only with one access rights you need to fix some error.
Please change the permissions to 0770 so that the directory cannot be listed by other users.
I have not checked further but i think you can change the falue to false in the configuration which will fix this issue, but as i have not yet done it (still in the process) there may be issues afterwards.
Interesting. I wonder how efficient it is with regards to unnecessary
metadata requests which Azure charges for. Ideally if you’re using blob you
would cache the metadata aggressively for that reason.
Interesting. I wonder how efficient it is with regards to unnecessary metadata requests which Azure charges for. Ideally if you’re using blob you would cache the metadata aggressively for that reason. …
On Wed, Jan 8, 2020 at 3:44 AM chapeau @.***> wrote: [image: AZUREBLOBNFS3] https://user-images.githubusercontent.com/19348103/71972009-1ca82780-320c-11ea-99aa-ab571c21f699.JPG It seems that it is possible to mount Blob natively via NFS now. Cheaper and easier than object storage (Blob, S3 or other) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2027?email_source=notifications&email_token=AAL7NADARSB62Z2HAHAUHQ3Q4WVATA5CNFSM4CVKCH32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIL6ZZQ#issuecomment-571993318>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAL7NAEVI6GUV7I4BDLQXBLQ4WVATANCNFSM4CVKCH3Q .
It is all about $$$ I guess, I use to have a 28 Millions requests per week on my S3 bucket for around 1000 files. In that case you just get the data and the metadata will be limited to files's GID/UID and embedded in the offering. Let's see .
would also be intersted. +1 Azure Blob with features like Soft Delete and a WORM Solution with Azure Storage Explorer, it would be very good product for the clients.
For Anyone who might be interested in getting this working, here are the steps with the Nextcloud docker container, adjust accordingly if you are running on bare metal
Install Blob fuse
As of this writing there is no blobfuse package available in the MS repo for debian 11, do the following to get this installed if using debian 11 (for other distros follow Blob fuse documentation) curl https://packages.microsoft.com/debian/10/prod/pool/main/b/blobfuse/blobfuse-1.4.3-Debian10.0-x86_64.deb --output blobfuse-1.4.3-Debian10.0-x86_64.deb sudo apt-get update sudo apt-get install fuse sudo dpkg -i blobfuse-1.4.3-Debian10.0-x86_64.deb
Create a mount folder for your nextcloud "/var/www/html" folder on host say /mnt/nextcloud
Create an empty folder say media inside this to be used as volume mounting point in container later, /mnt/nextcloud/media
Create a mount folder for your azure blob under this like/mnt/blob
Create a cache folder for blob fuse say /mnt/cache
Create a connection.cfg file say in /connection.cfg and add the following line to it (without the curly brackets) accountName {Azure Storage Acc Name} accountKey {Azure Storage Access Key} authType Key containerName {Azure Blob Container Name}
Run the following command to mount blob to dir sudo blobfuse /mnt/blob --tmp-path=/mnt/cache --config-file=/connection.cfg -o allow_other
Add the above command to fstab or a new systemd service to ensure this directory is mounted automatically after server reboots
On Docker:
Create the nextcloud container as documented in docker hub, just add a new volume to the above folder as follows docker run ... -v /mnt/nextcloud:/var/www/html -v /mnt/blob:/var/www/html/media ....
For Existing running containers, on portainer duplicate/recreate and add the volumes
Run Container, add the External Storage app to Nextcloud
Then mount the local folder /var/www/html/media to Nextcloud as external storage.
Enjoy!
P.S : Also tried to use Azure S3 gateway, Minio Azure Gateway etc, long story short, they dont work with Nextcloud External storage very well, this is the only method which worked for me 100% with good performance.
I was able to set up and use blobfuse pretty decently, only annoying thing I am unable to figure out yet is the fact that nextcloud goes on telling I am out of space since the blob storage is not reporting the storage space properly. It is showing and calculating the space that is left on the OS drive.
@brosahay Did you find any solution for the wrongly reported size? Does it influence the way Nextcloud works? Does it prevent uploading files to Nextcloud?
@brosahay Did you find any solution for the wrongly reported size? Does it influence the way Nextcloud works? Does it prevent uploading files to Nextcloud?
It does not affect the operations in any way as such. I am working on getting the blobfuse2 working right now.
Activity
MariusBluem commentedon Nov 6, 2016
Hm ... Let's put this as an enhancement. However - I do not see this in the near future 😁 ... Maybe you want to step in - or know somebody who wants so 😅
We have some external storage backends in Nextcloud, (Amazon) S3 seems to be finished for (primary) storage for Nextcloud 11: #768
This may be an alternative ... Sharepoint is also on the roadmap for 11, if you rely on Microsoft 😁
cc @icewind1991 @blizzz
pascalsimi commentedon Nov 7, 2016
I find it rather puzzling that blob is not supported.
If you look at the gartner quadrant then there is Amazon and Microsoft as the big players.
Supporting only one of them cuts down potential user base by approximately 50%. Take into account how many personal computers actually run on Microsoft it's even more puzzling to me why Microsoft blob is not on the roadmap. But hey its just my thoughts. Thanks for all the hard work you put into next cloud. All you guys are awesome!
[-]Feature Request : Integration with Azure blob storage[/-][+]Integration with Azure blob storage[/+]gjonespf commentedon Nov 7, 2016
Would love to see this also. May be worth mentioning that (and I haven't tested this) but Azure File Store allows SMB access, so you could potentially run Nextcloud on an Azure VM with external storage mapped to the Azure File Store, as long as the VM and storage were in the same region (otherwise you need SMB3 encryption support, which isn't there yet in Linux). Something I will have to investigate.
pascalsimi commentedon Nov 8, 2016
I have a few projects with Microsoft azure for processing and nextcloud for storage and sync right now and with their new push for openness on azure I think there is a good chance their evangelists might be interested in helping to implement that feature.
I'll mention it in the next meeting.
pilcherd commentedon Feb 9, 2017
Support for Azure Blob would be good. Interesting about Azure File Store and SMB. Will look into that.
MariusBluem commentedon Feb 22, 2017
https://github.com/icewind1991/files_external_azure
Digging for old code 😜😜 What was the state there @icewind1991 🤔🙈
icewind1991 commentedon Feb 22, 2017
Not working atm, will see if I can fix it when I get some time
strategywise commentedon May 24, 2017
We're also interested in this ability. If the SMB mapping on the host box works would it in theory at least be able to read/write to the blob? If I have to get a blob mounted on the VM and it works well I don't think that would be a very hard work around for the time being.
SpiraMirabilis commentedon Sep 16, 2017
I've been tinkering with this, using the S3 backend as a template. It's a bit of a work in progress. However, as a temporary measure you can use a Windows server (2008 or greater), with WebDAV authoring role for IIS and essentially act a proxy in between your users and azure blob storage, see this how-to: http://fabriccontroller.net/mounting-your-azure-file-shares-on-premises-through-webdav/ Edit: Actually, it looks like that only supports the Azure Files which is much more expensive than blob.
Should it support serving from the verizon/akamai CDN if it is linked to that storage account?
Also, either nextcloud will need dedicated access to the blob container, or maybe we can use a Service Bus subscription and just mandate that anyone adding/changing/removing a file triggers a message sent, that way we can confidently cache the metadata/contents and there wouldnt be polling which increases cost a lot.
MorrisJobke commentedon Dec 18, 2017
This should be definitely be implemented as an app - see the files_external_dropbox external storage for example: https://github.com/icewind1991/files_external_dropbox
7 remaining items
powareverb commentedon Sep 29, 2018
Thanks for the clarification @MorrisJobke is what I figured. You guys only have so much time in the day. I'll be looking into Minio on blob storage, as the s3 backend for Nextcloud seems more stable now. Might be worth looking at for others.
https://azuremarketplace.microsoft.com/en-us/marketplace/apps/minio.minioblob
ohthehugemanatee commentedon Oct 19, 2018
Thanks for the clarification.
For those who are curious, there are ways to mount Azure Blob storage directly on your VM or in k8s. That's what I ended up doing. Honestly if you're going to use it as primary storage, this is a better option than implementing in the application layer IMO.
@see
Hope this unblocks someone else in the future!
z2z commentedon Jan 28, 2019
Maybe in future, use of https://flysystem.thephpleague.com/docs/ it already has a lot of adapters.
mstenz commentedon Sep 16, 2019
also instead of BlobFUSE you could potentially map Azure Files with SMB directoy to the host vm.
But as you can mount it only with one access rights you need to fix some error.
f(\OC::$server->getConfig()->getSystemValue('check_data_directory_permissions', true) === false) {I have not checked further but i think you can change the falue to false in the configuration which will fix this issue, but as i have not yet done it (still in the process) there may be issues afterwards.
MourIdri commentedon Jan 8, 2020
It seems that it is possible to mount Blob natively via NFS now. Cheaper and easier than object storage (Blob, S3 or other)
SpiraMirabilis commentedon Jan 8, 2020
MourIdri commentedon Jan 8, 2020
It is all about $$$ I guess, I use to have a 28 Millions requests per week on my S3 bucket for around 1000 files. In that case you just get the data and the metadata will be limited to files's GID/UID and embedded in the offering. Let's see .
ThumbGen commentedon Apr 18, 2021
Did anyone tried the Azure Files NFS v3 as primary storage for NextCloud? Any hints/tips?
ajdinzutic commentedon May 17, 2021
would also be intersted. +1 Azure Blob with features like Soft Delete and a WORM Solution with Azure Storage Explorer, it would be very good product for the clients.
sudipmandal commentedon Mar 24, 2022
For Anyone who might be interested in getting this working, here are the steps with the Nextcloud docker container, adjust accordingly if you are running on bare metal
My setup:
Host : Debian 11, Docker 20.11
Nextcloud : Docker image 23.0.3
On Host:
As of this writing there is no blobfuse package available in the MS repo for debian 11, do the following to get this installed if using debian 11 (for other distros follow Blob fuse documentation)
curl https://packages.microsoft.com/debian/10/prod/pool/main/b/blobfuse/blobfuse-1.4.3-Debian10.0-x86_64.deb --output blobfuse-1.4.3-Debian10.0-x86_64.debsudo apt-get updatesudo apt-get install fusesudo dpkg -i blobfuse-1.4.3-Debian10.0-x86_64.debaccountName {Azure Storage Acc Name}accountKey {Azure Storage Access Key}authType KeycontainerName {Azure Blob Container Name}sudo blobfuse /mnt/blob --tmp-path=/mnt/cache --config-file=/connection.cfg -o allow_otherOn Docker:
Create the nextcloud container as documented in docker hub, just add a new volume to the above folder as follows
docker run ... -v /mnt/nextcloud:/var/www/html -v /mnt/blob:/var/www/html/media ....For Existing running containers, on portainer duplicate/recreate and add the volumes
Run Container, add the External Storage app to Nextcloud
Then mount the local folder
/var/www/html/mediato Nextcloud as external storage.Enjoy!
P.S : Also tried to use Azure S3 gateway, Minio Azure Gateway etc, long story short, they dont work with Nextcloud External storage very well, this is the only method which worked for me 100% with good performance.
brosahay commentedon Nov 2, 2022
I was able to set up and use blobfuse pretty decently, only annoying thing I am unable to figure out yet is the fact that nextcloud goes on telling I am out of space since the blob storage is not reporting the storage space properly. It is showing and calculating the space that is left on the OS drive.
ThumbGen commentedon Dec 16, 2022
@brosahay Did you find any solution for the wrongly reported size? Does it influence the way Nextcloud works? Does it prevent uploading files to Nextcloud?
z2z commentedon Jan 2, 2023
@brosahay @ThumbGen report at https://github.com/Azure/azure-storage-fuse/issues
Also jfyi, both sftp and nfsv3 have cost. https://azure.microsoft.com/en-us/pricing/details/storage/blobs/
brosahay commentedon Jan 19, 2023
It does not affect the operations in any way as such. I am working on getting the blobfuse2 working right now.