Commit Graph

127 Commits

Author SHA1 Message Date
Roeland Jago Douma d5850eb28f
Merge pull request #18955 from adrb/swift_upload_large_objects
Large Object support for OpenStack Swift
2020-05-06 13:03:24 +02:00
Robin Appelman 4e6d3d1880
Fix getDirectoryContent implementation for Jail wrapper
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-30 15:27:09 +02:00
Roeland Jago Douma f6b435d0d9
Run the cs fixer for green ci
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-04-30 10:28:44 +02:00
Roeland Jago Douma a1c1b354fc
Merge pull request #18883 from adrb/optimize_swift_14116
Optimize Openstack Swift files download
2020-04-30 09:06:19 +02:00
Christoph Wurst cb057829f7
Update license headers for 19
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-29 11:57:22 +02:00
Adrian Brzezinski e107519295 Large Object support for OpenStack Swift.
Until now, you wouldn't be able to create
objects larger that 5GB.

It's somewhat related with pull #18883

Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
2020-04-27 14:23:59 +02:00
Adrian Brzezinski 354c0996d8 It's a fix for issue #14116,
Improves efficiency when downloading files from Swift storage.
Before, files were downloaded and then pushed back to user.
That behaevior causes all kinds of performance problems.

Now, files are streamed directly to user.

Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
2020-04-27 14:16:50 +02:00
Robin Appelman 9735b5d61f
Add method to storage backends to get directory content with metadata
Currently you need to use `opendir` and then call `getMetadata` for
every file, which adds overhead because most storage backends already
get the metadata when doing the `opendir`.

While storagebackends can (and do) use caching to relief this problem,
this adds cache invalidation dificulties and only a limited number of
items are generally cached (to prevent memory usage exploding when
scanning large storages)

With this new methods storage backends can use the child metadata they
got from listing the folder to return metadata without having to keep
seperate caches.

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-20 15:45:04 +02:00
Julien Lutran 3e849daf11
Fix issues with Keystone auth v3 in files_external app
Signed-off-by: Julien Lutran <julien.lutran@corp.ovh.com>
2020-04-15 10:33:08 +02:00
Christoph Wurst 734c62bee0
Format code according to PSR2
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:56:50 +02:00
Christoph Wurst 28f8eb5dba
Add visibility to all constants
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:54:27 +02:00
Christoph Wurst 1584c9ae9c
Add visibility to all methods and position of static keyword
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:51:06 +02:00
Christoph Wurst caff1023ea
Format control structures, classes, methods and function
To continue this formatting madness, here's a tiny patch that adds
unified formatting for control structures like if and loops as well as
classes, their methods and anonymous functions. This basically forces
the constructs to start on the same line. This is not exactly what PSR2
wants, but I think we can have a few exceptions with "our" style. The
starting of braces on the same line is pracrically standard for our
code.

This also removes and empty lines from method/function bodies at the
beginning and end.

Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 14:19:56 +02:00
Christoph Wurst 14c996d982
Use elseif instead of else if
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 10:35:09 +02:00
Christoph Wurst 44577e4345
Remove trailing and in between spaces
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 16:07:47 +02:00
Christoph Wurst afbd9c4e6e
Unify function spacing to PSR2 recommendation
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 13:54:22 +02:00
Christoph Wurst 41b5e5923a
Use exactly one empty line after the namespace declaration
For PSR2

Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 11:48:10 +02:00
Robin Appelman 7b07e7251c
make seekable s3 stream generic
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 15:21:05 +02:00
Lukas Stabe 14401efb0f
Enable fseek for files in S3 storage
Signed-off-by: Lukas Stabe <lukas@stabe.de>
2020-04-01 15:21:01 +02:00
Christoph Wurst 1a9330cd69
Update the license headers for Nextcloud 19
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-31 14:52:54 +02:00
Christoph Wurst b80ebc9674
Use the short array syntax, everywhere
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-26 16:34:56 +01:00
Roeland Jago Douma 9a35e789b4
Get correct mimetype on objectstores
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-03-11 11:14:12 +01:00
Christoph Wurst 5bf3d1bb38
Update license headers
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2019-12-05 15:38:45 +01:00
Roeland Jago Douma 68748d4f85
Some php-cs fixes
* Order the imports
* No leading slash on imports
* Empty line before namespace
* One line per import
* Empty after imports
* Emmpty line at bottom of file

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-11-22 20:52:10 +01:00
Robin Appelman 733d4b6cca
dont delete cache entries if deleting an object from object store failed
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-09-25 18:09:45 +02:00
Morris Jobke e45fb5fa3e
Fix typo in comment
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2019-07-29 14:55:43 +02:00
Marcel Klehr d46744e2f1 Fix File#putContents(string) on ObjectStorage
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
2019-07-17 14:58:56 +02:00
Roeland Jago Douma e953205908
Use HTTP1.1 to read S3 objects
Some of the READs otherwise use HTTP/1.0 which is not always supported
by all backends. HTTP/1.1 is there since 1999 way longer than S3 so safe
to assume it is always there IMO.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-07-10 11:42:22 +02:00
Morris Jobke 96e892770d
Merge pull request #15715 from nextcloud/s3-bucket-name
Allow other code to use the s3 client from object store
2019-05-24 15:03:22 +02:00
Robin Appelman 834db3af2c
add method to get bucket name from s3 storage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-05-24 14:02:03 +02:00
Robin Appelman 06a119242e
make ObjectStoreStorage::getURN public
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-05-22 12:17:33 +02:00
Robin Appelman 439b341e89
add method to get the used object store from the storage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-05-21 17:18:00 +02:00
Roeland Jago Douma dedb84acb0
Always use a temp file for swift writes
Apparently the if statement doesn't work in all cases (even if I could
not reproduce it). So for the time being we will just not directly
stream to swift.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-04-18 21:04:27 +02:00
Roeland Jago Douma 1b27e9578d
Merge pull request #14424 from nextcloud/fix/13554/swift_to_tmp
Use a tmp file for swift writes
2019-03-06 15:33:15 +01:00
Morris Jobke debd32b461
Merge pull request #14175 from skydiablo/master
Fix ObjectStorage expired token (v3)
2019-03-05 18:59:45 +01:00
Roeland Jago Douma 1153123b90
Use a tmp file for swift writes
Else this leads to a seekable stream error with chunked uploads

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-03-04 13:11:59 +01:00
Roeland Jago Douma df55781814
Merge pull request #14254 from sparrowjack63/master
Dont check Bucket Name in Nextcloud
2019-03-01 13:46:04 +01:00
S. Cat e3142a91b2 Bucket name - Alert in debug mode
Alert in debug mode only when trying to mount non compatible DNS bucket name (in order to not flood the logs)

Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +01:00
S. Cat 381ae44d38 Correct code indent
Just correct code indent.

Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +01:00
S. Cat 77473b0228 Warning on S3 not compatible DNS bucket name
The new created bucket should respect the DNS compatibility, nevertheless, Nextcloud should accept to mount "old created" buckets that does not respect DNS compatibility (Backward compatibility, or compatibility with CEPH).

Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +01:00
Sebastien Cat 59c3d8d85f Dont check Bucket Name in Nextcloud
Signed-off-by: Sebastien Cat <sparrow.jack63@gmail.com>
Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +01:00
Roeland Jago Douma f6f002e2ad
Merge pull request #14210 from nextcloud/fix/14192/fix_empty_uploads
Fix empty file uploads to S3 (and other streaming storages)
2019-02-18 15:39:17 +01:00
Roeland Jago Douma 104616c1da
Use CountWrapper instead of almost identical CountReadStream
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-02-15 13:24:58 +01:00
Roeland Jago Douma 61d66d7954
Fix empty file uploads to S3
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-02-15 08:41:39 +01:00
Volker d06c25b9c8
Update SwiftFactory.php
Auth-Service (in explizit v3) will recheck the cached-token and will end in an "token expired exception".
2019-02-13 12:26:27 +01:00
William Pain c26bdd4731 Fix: Check if `$this->params['user']` is an array
Signed-off-by: William Pain <pain.william@gmail.com>
2019-02-05 09:56:42 +01:00
Robin Appelman 1d322d3b37
always use multipart uploader for s3 uploads
the multipart uploader handles non seekable streams while `upload` does not

Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-01-28 14:51:19 +01:00
Morris Jobke 5a27e54f4b
Merge pull request #13032 from nextcloud/objectstore-write-exists
upload new files in objectstore to a .part path first
2019-01-14 11:26:29 +01:00
Robin Appelman 615fb8cd77
Cache tokens when using swift's v2 authentication
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-01-13 19:59:48 +01:00
Robin Appelman d6bf5d4384
upload new files in objectstore to a .part path first
This prevent the object store and cache from getting out of sync
when an objectstore silently fails or the php process get's killed
during the upload without giving us the chance to cleanup

Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-12-12 15:24:40 +01:00