Commit Graph

682 Commits

Author SHA1 Message Date
Christoph Wurst 9d392891be
Disable Client-Side Monitoring on AWS storage
The S3 client enables this by default and then tries to read
`.aws/config`. This causes `open_basedir` restriction related error for
some setups. So this patch disables the CSM because it's most likely
unused anyway.

Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-06-15 09:13:08 +02:00
Roeland Jago Douma 8bee8e1317
Merge pull request #21317 from FlorentCoppint/s3_upload_part_size
Upload part size as S3 parameter instead of constant value
2020-06-09 19:28:58 +02:00
Florent 3594ba6971 Upload part size as S3 parameter instead of constant value
Some S3 providers need a custom upload part size (500 MB static value in Nextcloud).
Here is a commit to change this value via S3 configuration, instead of using S3_UPLOAD_PART_SIZE constant.
A new parameter is added for an S3 connection : uploadPartSize

Signed-off-by: Florent <florent@coppint.com>
2020-06-09 09:18:42 +02:00
Robin Appelman e75797ad67
increase log level for storage self-test
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-06-08 18:12:49 +02:00
Roeland Jago Douma 6aa6ab3e02
Add lazy events for the Node API
Right now if you want to get events via the Node API you have to have a
real instance of the Root. Which in turns sets up the whole FS.

We should make sure this is done lazy. Else enabling the preview
generator for example makes you setup the whole FS on each and every
authenticated call.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-05-28 12:35:45 +02:00
Roeland Jago Douma 6b26744787
Clear the statscache before fetching the metadata
Else if a lot of writes happen. It might happen that an old stat result
is used. Resulting in a wrong file size for the file. For example the
text app when a lot of people edit at the same time.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-05-22 09:21:53 +02:00
Robin Appelman f7153a3d0c
generalize the "lazy folder" concept
makes it easy to use lazy intialized folder in other places

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-05-11 15:36:13 +02:00
Roeland Jago Douma d5850eb28f
Merge pull request #18955 from adrb/swift_upload_large_objects
Large Object support for OpenStack Swift
2020-05-06 13:03:24 +02:00
Robin Appelman 4e6d3d1880
Fix getDirectoryContent implementation for Jail wrapper
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-30 15:27:09 +02:00
Roeland Jago Douma f6b435d0d9
Run the cs fixer for green ci
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-04-30 10:28:44 +02:00
Roeland Jago Douma a1c1b354fc
Merge pull request #18883 from adrb/optimize_swift_14116
Optimize Openstack Swift files download
2020-04-30 09:06:19 +02:00
Christoph Wurst cb057829f7
Update license headers for 19
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-29 11:57:22 +02:00
Adrian Brzezinski e107519295 Large Object support for OpenStack Swift.
Until now, you wouldn't be able to create
objects larger that 5GB.

It's somewhat related with pull #18883

Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
2020-04-27 14:23:59 +02:00
Adrian Brzezinski 354c0996d8 It's a fix for issue #14116,
Improves efficiency when downloading files from Swift storage.
Before, files were downloaded and then pushed back to user.
That behaevior causes all kinds of performance problems.

Now, files are streamed directly to user.

Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
2020-04-27 14:16:50 +02:00
Robin Appelman 436908ee3f
filter out blacklisted files in getDirectoryContent
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-23 13:34:55 +02:00
Robin Appelman 9735b5d61f
Add method to storage backends to get directory content with metadata
Currently you need to use `opendir` and then call `getMetadata` for
every file, which adds overhead because most storage backends already
get the metadata when doing the `opendir`.

While storagebackends can (and do) use caching to relief this problem,
this adds cache invalidation dificulties and only a limited number of
items are generally cached (to prevent memory usage exploding when
scanning large storages)

With this new methods storage backends can use the child metadata they
got from listing the folder to return metadata without having to keep
seperate caches.

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-20 15:45:04 +02:00
Roeland Jago Douma 79cf13e349
Merge pull request #16946 from exussum12/patch-1
Increase max read
2020-04-16 10:37:10 +02:00
Scott Dutton 65e134c13a
Update View.php 2020-04-15 19:10:06 +01:00
Morris Jobke 7fe8477830
Run cs:fix and fix HashWrapper code
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2020-04-15 14:08:56 +02:00
Morris Jobke d8557d0d8a
Merge pull request #19540 from jlutran/fixKeystoneV3Auth
Fix issues with Keystone auth v3 in files_external app
2020-04-15 14:05:32 +02:00
Christoph Wurst 301327135c
Merge pull request #20361 from nextcloud/write-stream-close-on-exception
Close the streams in `writeStream` even when there is an exception
2020-04-15 10:48:24 +02:00
Julien Lutran 3e849daf11
Fix issues with Keystone auth v3 in files_external app
Signed-off-by: Julien Lutran <julien.lutran@corp.ovh.com>
2020-04-15 10:33:08 +02:00
Robin Appelman c2677682c4
Return hashes of uploaded content for dav uploads
hashes are set in "X-Hash-MD5", "X-Hash-SHA1" and "X-Hash-SHA256" headers.

these headers are set for file uploads and the MOVE request at the end of a multipart upload.

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-13 12:57:27 +02:00
Scott Dutton 75b99e85ff
Increase max read
8kb is very low, especially given this will be local files

Signed-off-by: Scott Dutton <scott@exussum.co.uk>
2020-04-12 13:01:29 +01:00
Christoph Wurst 734c62bee0
Format code according to PSR2
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:56:50 +02:00
Christoph Wurst 28f8eb5dba
Add visibility to all constants
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:54:27 +02:00
Christoph Wurst 1584c9ae9c
Add visibility to all methods and position of static keyword
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:51:06 +02:00
Christoph Wurst a7c8d26d31
Add visibility to all properties and move static keyword
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:48:31 +02:00
Christoph Wurst caff1023ea
Format control structures, classes, methods and function
To continue this formatting madness, here's a tiny patch that adds
unified formatting for control structures like if and loops as well as
classes, their methods and anonymous functions. This basically forces
the constructs to start on the same line. This is not exactly what PSR2
wants, but I think we can have a few exceptions with "our" style. The
starting of braces on the same line is pracrically standard for our
code.

This also removes and empty lines from method/function bodies at the
beginning and end.

Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 14:19:56 +02:00
Christoph Wurst 14c996d982
Use elseif instead of else if
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 10:35:09 +02:00
Roeland Jago Douma fb64fda4d8
Merge pull request #20385 from nextcloud/skeeable-http-harden
harden seekable http stream a bit against failures
2020-04-10 08:34:52 +02:00
Christoph Wurst 44577e4345
Remove trailing and in between spaces
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 16:07:47 +02:00
Christoph Wurst 42625a46be
Remove spaces after method or function call
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 16:05:56 +02:00
Robin Appelman 60de74ac40
harden seekable http stream a bit against failures
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-09 14:37:01 +02:00
Christoph Wurst afbd9c4e6e
Unify function spacing to PSR2 recommendation
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 13:54:22 +02:00
Christoph Wurst 41b5e5923a
Use exactly one empty line after the namespace declaration
For PSR2

Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 11:48:10 +02:00
Robin Appelman ccbf3059ba
Close the streams in `writeStream` even when there is an exception
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-08 16:40:56 +02:00
Joas Schilling f8a5812f98
Try to use the display name of file transfers
Signed-off-by: Joas Schilling <coding@schilljs.com>
2020-04-08 15:41:20 +02:00
Roeland Jago Douma f5919d5b83
Merge pull request #20033 from nextcloud/s3-seekable-stream
Enable fseek for files in S3 storage
2020-04-06 21:38:49 +02:00
Julius Härtl 56aa8fd035
Properly hint exceptions thrown by getUserFolder
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2020-04-03 23:06:26 +02:00
Robin Appelman e22a28ecc2
update autoloader
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-03 16:27:20 +02:00
Robin Appelman 3ba46f3b50
add basic tests for s3 seeking and add some error handling if reopen return the wrong range
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 15:23:05 +02:00
Robin Appelman 7b07e7251c
make seekable s3 stream generic
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 15:21:05 +02:00
Lukas Stabe 14401efb0f
Enable fseek for files in S3 storage
Signed-off-by: Lukas Stabe <lukas@stabe.de>
2020-04-01 15:21:01 +02:00
Robin Appelman d9184584e0
restart transaction when cache insert fails due to duplicate
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:17:34 +02:00
Robin Appelman 5199f80196
call cache->insert directly instead of going trough cache->put first when scanning
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:17:28 +02:00
Robin Appelman 9c7da26394
dont try to get existing cache data during scanning if we already know its a new file
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:17:25 +02:00
Robin Appelman 3fb32f6f3d
reuse known parentId when inserting cache entries in scanner
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:14:52 +02:00
Robin Appelman 5439469b68
optimize getMetaData for local storage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:14:48 +02:00
Christoph Wurst 1a9330cd69
Update the license headers for Nextcloud 19
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-31 14:52:54 +02:00